diff --git a/source/_posts/浏览器端crc32文件计算以及分片上传思路.md b/source/_posts/浏览器端crc32文件计算以及分片上传思路.md new file mode 100644 index 0000000..c903ae2 --- /dev/null +++ b/source/_posts/浏览器端crc32文件计算以及分片上传思路.md @@ -0,0 +1,133 @@ +--- +title: 浏览器端crc32文件计算以及分片上传思路 +id: '362' +cover: '/images/AAAABSGsf8u86EvIjRhj.jpg' +categories: + - 分享 +tags: + - JavaScript + - html + - crc32 + - 浏览器 + - 分片上传 +date: 2020-12-27 15:21:00 +--- + +大文件上传我们知道需要做MD5计算以及文件分片,由于最近业务业务上的调整,需要把MD5换成crc32,,crc32可以校验文件的完整性,对于业务处理比较方便。这里记录一下解决思路吧。 + +# 1. 文件分片 + +考虑到文件可能很大,出于性能的考虑,优先把文件分片,这里我设置默认5M大小一个分片: + +``` js +const createFileChunk = (file, size = 5 * 1024 * 1024) => { + let chunks = []; + let cur = 0; + while (cur < file.size) { + chunks.push(file.slice(cur, cur + size)); + cur += size; + } + return chunks; +}; +``` + +# 2. 文件crc32计算 + +首先找一个crc32的库,这里我选了sheetjs的[crc32](https://github.com/SheetJS/js-crc32)库。 +为了提高性能,这种纯计算的功能使用web workers另起一个线程处理。 + +``` js +// crc32_worker.js + +// 引入crc32.js库,importScripts只支持url引入 +self.importScripts("https://cdn.bootcdn.net/ajax/libs/crc-32/1.2.0/crc32.js"); + +// 生成文件 hash +self.onmessage = e => { + const { + fileChunks + } = e.data; + let crc32 = null; + let percentage = 0; + let count = 0; + const loadNext = index => { + const reader = new FileReader(); + reader.readAsArrayBuffer(fileChunks[index]); + reader.onload = e => { + count++; + const blob = new Uint8Array(e.target.result); + if (index === 0) { + crc32 = self.CRC32.buf(blob); + } else { + crc32 = self.CRC32.buf(blob, crc32); + } + + if (count === fileChunks.length) { + self.postMessage({ + percentage: 100, + hash: crc32 + }); + self.close(); + } else { + percentage += 100 / fileChunks.length; + self.postMessage({ + percentage + }); + loadNext(count); + } + }; + }; + loadNext(0); +}; +``` + +web workers调用以及封装: + +``` js +// 计算文件 crc32(web-worker)参数为上面分片后返回的chunks +const createFileCRC32InWorker = fileChunks => { + return new Promise(resolve => { + // 引入上面的worker文件 + const worker = new Worker("./crc32_worker.js"); + worker.postMessage({ + fileChunks + }); + worker.onmessage = e => { + // 这边加了进度条 这里的进度条,看需要显示 + const { + percentage, + hash + } = e.data; + hash && resolve(hash); + }; + }); +}; +``` + +# 3. 上传进度本地缓存 + +计算出的crc32值用来作为判断文件的key,可在localStorage缓存分片上传的进度。 + +``` js +const uploadCache = { + 1259060791: { + position: [0, 0, 0] + } +} +``` + +如上代码, ` 1259060791 ` 为文件的crc32值,把它作为key, ` position ` 为上传的进度,比如一个15M的文件,会分成3片(默认设置了每片5M),然后根据自己的上传业务,上传完成的分片标记为 ` 1 ` 。 + +``` js +const uploadCache = { + 1259060791: { + position: [1, 0, 0] // 第一片上传完成后标记为1 + } +} +``` + +值得注意的是,通常分片上传都会多个分片并行上传,标记为 ` 1 ` 需要留意 ` position ` 的下标跟分片的下标是否一致。 + +# 4. 浏览器刷新后获取文件的问题 + +由于浏览器的安全机制,刷新后无法自动获取到原来的文件,所以需要通过提示去引导用户重现选择该文件,然后才可以断点续传。 diff --git a/source/images/AAAABSGsf8u86EvIjRhj.jpg b/source/images/AAAABSGsf8u86EvIjRhj.jpg new file mode 100644 index 0000000..2efd57e Binary files /dev/null and b/source/images/AAAABSGsf8u86EvIjRhj.jpg differ diff --git a/source/images/v2-0dff7a84fe57e0d911f6efd2ba445db5_720w.jpg b/source/images/config/v2-0dff7a84fe57e0d911f6efd2ba445db5_720w.jpg similarity index 100% rename from source/images/v2-0dff7a84fe57e0d911f6efd2ba445db5_720w.jpg rename to source/images/config/v2-0dff7a84fe57e0d911f6efd2ba445db5_720w.jpg diff --git a/source/images/v2-1497fa6a5de2596909dc2af7a6233e37_1440w.jpg b/source/images/config/v2-1497fa6a5de2596909dc2af7a6233e37_1440w.jpg similarity index 100% rename from source/images/v2-1497fa6a5de2596909dc2af7a6233e37_1440w.jpg rename to source/images/config/v2-1497fa6a5de2596909dc2af7a6233e37_1440w.jpg diff --git a/source/images/v2-1ff0597480e2ef8999a2bc86a69f6a3e_r.jpg b/source/images/config/v2-1ff0597480e2ef8999a2bc86a69f6a3e_r.jpg similarity index 100% rename from source/images/v2-1ff0597480e2ef8999a2bc86a69f6a3e_r.jpg rename to source/images/config/v2-1ff0597480e2ef8999a2bc86a69f6a3e_r.jpg diff --git a/source/images/v2-498e45cff82e401ae43378031b8fc740_1440w.jpg b/source/images/config/v2-498e45cff82e401ae43378031b8fc740_1440w.jpg similarity index 100% rename from source/images/v2-498e45cff82e401ae43378031b8fc740_1440w.jpg rename to source/images/config/v2-498e45cff82e401ae43378031b8fc740_1440w.jpg diff --git a/source/images/v2-498e45cff82e401ae43378031b8fc740_256w.jpg b/source/images/config/v2-498e45cff82e401ae43378031b8fc740_256w.jpg similarity index 100% rename from source/images/v2-498e45cff82e401ae43378031b8fc740_256w.jpg rename to source/images/config/v2-498e45cff82e401ae43378031b8fc740_256w.jpg diff --git a/source/images/v2-5d580945ed25ce42f2fa5bea4cb0d278_r.jpg b/source/images/config/v2-5d580945ed25ce42f2fa5bea4cb0d278_r.jpg similarity index 100% rename from source/images/v2-5d580945ed25ce42f2fa5bea4cb0d278_r.jpg rename to source/images/config/v2-5d580945ed25ce42f2fa5bea4cb0d278_r.jpg diff --git a/source/images/v2-71c2d3e28a607d4732580d5fa8fcbac7_r.jpg b/source/images/config/v2-71c2d3e28a607d4732580d5fa8fcbac7_r.jpg similarity index 100% rename from source/images/v2-71c2d3e28a607d4732580d5fa8fcbac7_r.jpg rename to source/images/config/v2-71c2d3e28a607d4732580d5fa8fcbac7_r.jpg diff --git a/source/images/v2-8336cecd23d2a4134033791a8e5211fa_720w.jpg b/source/images/config/v2-8336cecd23d2a4134033791a8e5211fa_720w.jpg similarity index 100% rename from source/images/v2-8336cecd23d2a4134033791a8e5211fa_720w.jpg rename to source/images/config/v2-8336cecd23d2a4134033791a8e5211fa_720w.jpg diff --git a/source/images/v2-94c8508fb5f0acc59dd3616512a6cf47_720w.jpg b/source/images/config/v2-94c8508fb5f0acc59dd3616512a6cf47_720w.jpg similarity index 100% rename from source/images/v2-94c8508fb5f0acc59dd3616512a6cf47_720w.jpg rename to source/images/config/v2-94c8508fb5f0acc59dd3616512a6cf47_720w.jpg diff --git a/source/images/v2-be9ee0a7eef1372e6ff9891436c61389_hd.gif b/source/images/config/v2-be9ee0a7eef1372e6ff9891436c61389_hd.gif similarity index 100% rename from source/images/v2-be9ee0a7eef1372e6ff9891436c61389_hd.gif rename to source/images/config/v2-be9ee0a7eef1372e6ff9891436c61389_hd.gif diff --git a/source/images/v2-d1ccc3ef1cec99e3ae06d300c5ca0f7c_720w.jpg b/source/images/config/v2-d1ccc3ef1cec99e3ae06d300c5ca0f7c_720w.jpg similarity index 100% rename from source/images/v2-d1ccc3ef1cec99e3ae06d300c5ca0f7c_720w.jpg rename to source/images/config/v2-d1ccc3ef1cec99e3ae06d300c5ca0f7c_720w.jpg diff --git a/source/images/v2-e6df90886acf0bfc934f2902cec54143_720w.jpg b/source/images/config/v2-e6df90886acf0bfc934f2902cec54143_720w.jpg similarity index 100% rename from source/images/v2-e6df90886acf0bfc934f2902cec54143_720w.jpg rename to source/images/config/v2-e6df90886acf0bfc934f2902cec54143_720w.jpg diff --git a/source/images/v2-f119c1dab2d95d62b0808f3b334dd8a1_720w.jpg b/source/images/config/v2-f119c1dab2d95d62b0808f3b334dd8a1_720w.jpg similarity index 100% rename from source/images/v2-f119c1dab2d95d62b0808f3b334dd8a1_720w.jpg rename to source/images/config/v2-f119c1dab2d95d62b0808f3b334dd8a1_720w.jpg diff --git a/themes/butterfly/_config.yml b/themes/butterfly/_config.yml index 5895a7a..097be30 100644 --- a/themes/butterfly/_config.yml +++ b/themes/butterfly/_config.yml @@ -83,14 +83,14 @@ favicon: /img/favicon.png # Avatar (頭像) avatar: - img: /images/v2-498e45cff82e401ae43378031b8fc740_256w.jpg + img: /images/config/v2-498e45cff82e401ae43378031b8fc740_256w.jpg effect: false # The banner image of home page index_img: # If the banner of page not setting, it will show the top_img -default_top_img: /images/v2-1ff0597480e2ef8999a2bc86a69f6a3e_r.jpg +default_top_img: /images/config/v2-1ff0597480e2ef8999a2bc86a69f6a3e_r.jpg # The banner image of archive page archive_img: @@ -124,15 +124,15 @@ cover: # When cover is not set, the default cover is displayed (當沒有設置cover時,默認的封面顯示) default_cover: # - https://cdn.jsdelivr.net/gh/jerryc127/CDN@latest/cover/default_bg.png - - /images/v2-0dff7a84fe57e0d911f6efd2ba445db5_720w.jpg - - /images/v2-5d580945ed25ce42f2fa5bea4cb0d278_r.jpg - - /images/v2-71c2d3e28a607d4732580d5fa8fcbac7_r.jpg - - /images/v2-94c8508fb5f0acc59dd3616512a6cf47_720w.jpg - - /images/v2-8336cecd23d2a4134033791a8e5211fa_720w.jpg - - /images/v2-be9ee0a7eef1372e6ff9891436c61389_hd.gif - - /images/v2-d1ccc3ef1cec99e3ae06d300c5ca0f7c_720w.jpg - - /images/v2-e6df90886acf0bfc934f2902cec54143_720w.jpg - - /images/v2-f119c1dab2d95d62b0808f3b334dd8a1_720w.jpg + - /images/config/v2-0dff7a84fe57e0d911f6efd2ba445db5_720w.jpg + - /images/config/v2-5d580945ed25ce42f2fa5bea4cb0d278_r.jpg + - /images/config/v2-71c2d3e28a607d4732580d5fa8fcbac7_r.jpg + - /images/config/v2-94c8508fb5f0acc59dd3616512a6cf47_720w.jpg + - /images/config/v2-8336cecd23d2a4134033791a8e5211fa_720w.jpg + - /images/config/v2-be9ee0a7eef1372e6ff9891436c61389_hd.gif + - /images/config/v2-d1ccc3ef1cec99e3ae06d300c5ca0f7c_720w.jpg + - /images/config/v2-e6df90886acf0bfc934f2902cec54143_720w.jpg + - /images/config/v2-f119c1dab2d95d62b0808f3b334dd8a1_720w.jpg # Replace Broken Images (替換無法顯示的圖片) error_img: