“It is a sign that people aren’t happy within the U.S. government, clearly. The shooting [of Good] was the last straw for many people," Dominick Skinner, ICE list founder, told The Beast.

Information included in the new leak includes around "1,800 on-the-ground agents and 150 supervisors. Early analysis by the organization suggests that around 80 per cent of the staff identified remain employed by DHS," according to The Beast.

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 9 months ago (6 children)

Any chance you know Python, or maybe you can have ChatGPT write a script that uses bs4 to parse through each of the name hyperlinks in the html? A simple loop should be able to visit each page, cache the results on disk, and finish up eventually.

The link doesn’t work where I am, still.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 0 points 9 months ago* (last edited 9 months ago) (5 children)

    it's very slow for me. i'm looking into python and all the other stuff. if you one specifically that will scrap the whole site, i'm down to run it. any OS except mac i can use. i can't copy/paste each agent page, looking to scrap the whole site at once

    edit!

    i think i have it scraping it all to html files. it's taking time, just trying to make sure it gets the agents and incidents at least.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 1 point 9 months ago (4 children)

    I see the edit, sweet!

    I wouldn’t be able to write the script myself because I can’t load the site at all. So I can’t analyze the html and determine how to loop through it an extract all the additional links for scraping. Unless you want to send me the html. Then I can work with that, but without testing on the live site.

    If you have something working though, great! Let’s just start there.

    Check by opening one of the html files with your browser, or VSCode or something, and see if it has the data you were expecting.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 0 points 9 months ago (3 children)

    the files look correct, just html version of the pages, just not sure if it's going to crawl and get every page. i don't even are if it's just a folder of html file sat this time, as long as all the agents are retrived. will keep testing.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 9 months ago (2 children)

    I just set up a little bot in Jupyter Notebook to get them as well, and it’s working! Very slow, but working. I just fetched all the links to the agent detail pages. There are 1532 total. Once I fetch all the detail pages, I’ll move on to the incidents too.

    Definitely keep pulling yourself, though. Who knows if I will be able to get them all?

    Anything else worth grabbing while I can?

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 9 months ago* (1 child)

    not sure, i'm looking through agents now. it's HTML, and loads with broken CSS/styles, but the text is there

    we'll have more once the site is fully working. maybe the admin will allow ZIN backup downloads, make it easier to mirror. i don't htink he expected to get this much traction.

  • source
  • parent
  • hideshow 1 child comment