Hello [email protected]

I intend to upload an archive of a website I scraped for the last few weeks. The torrent is about 4 TB large and contains 400000 folders which in turn contain 40 .jpg’s on average (plus some metadata).

Should I just create the torrent with the files as they are right now, or should I put the individual folders in archives (or maybe even the entire torrent?)?

Thanks in advance

  • fkn@lemmy.world
    link
    fedilink
    English
    arrow-up
    40
    arrow-down
    1
    ·
    1 year ago

    People will want to download chicks or parts of the torrent, you should leave them separate.

    • cesium@sh.itjust.worksOP
      link
      fedilink
      English
      arrow-up
      5
      ·
      1 year ago

      Makes sense. What about individual folders? I’m concerned that the massive amount of small files could slow down the download quite a bit. Unfortunately, I’m not too familiar with the BitTorrent protocol, so I don’t know if this would have an impact.

      • zodo123@lemmy.world
        link
        fedilink
        English
        arrow-up
        14
        ·
        1 year ago

        Small files don’t impact BitTorrent transfers. They’re sent in pieces of a fixed size, and those pieces can contain multiple files.

      • fkn@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        Depends on final size of the folder zips. 20-30mb? Sure. 200-300? Nah.

  • TheCraiggers@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    15
    ·
    1 year ago

    Jpeg is already compressed, so compressing them again won’t do anything but make it impossible for people to selectively download just the image/folder they desire.

    Metadata on the other hand sounds like text files, which compress very well. Wether the space savings is worth it is hard to answer without more info. I’d personally lean towards not archiving it.

  • UntouchedWagons@lemmy.ca
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    Leave them as is. I hate having to deal with archives inside torrents unless there’s a very good reason for them.

  • free@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    arrow-down
    9
    ·
    1 year ago

    wow, dedication bud, few weeks to web rip. well done. true pirate here. tx 4 ur service. I’d love to know which site. 😛 feel free to DM me. I’m curious.