{"id":991926,"date":"2025-05-15T09:47:20","date_gmt":"2025-05-15T09:47:20","guid":{"rendered":"https:\/\/mynavitechtus.com\/?p=991926"},"modified":"2025-07-25T03:32:51","modified_gmt":"2025-07-25T03:32:51","slug":"experience-handling-large-csv-uploads-5gb-with-complex-validation-2","status":"publish","type":"post","link":"https:\/\/mynavitechtus.com\/ja\/experience-handling-large-csv-uploads-5gb-with-complex-validation-2\/","title":{"rendered":"\u8907\u96d1\u306a\u691c\u8a3c\u3092\u4f34\u3046\u5927\u898f\u6a21\u306aCSV\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\uff085GB\uff09\u306e\u51e6\u7406\u7d4c\u9a13"},"content":{"rendered":"
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u5fc5\u8981\u6761\u4ef6<\/span><\/h3>
  • \u30e6\u30fc\u30b6\u30fc\u304c\u6700\u59275GB\u306eCSV\u30d5\u30a1\u30a4\u30eb\u3092\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u3067\u304d\u308b\u3088\u3046\u306b\u3059\u308b\u3053\u3068\u3002\u00a0<\/span>
    <\/span><\/li>
  • \u30c7\u30fc\u30bf\u3092\u691c\u8a3c\u3057\u3001\u30a8\u30e9\u30fc\u30ed\u30a6\u3092\u3059\u3079\u3066\u30ed\u30b0\u306b\u8a18\u9332\u3059\u308b\u3053\u3068\uff1a<\/span>
    <\/span>
    • \u5217\u306e\u9577\u3055\u304c200\u6587\u5b57\u3092\u8d85\u3048\u308b\u5834\u5408\u3002<\/span>
      <\/span><\/li>
    • \u8a31\u53ef\u3055\u308c\u3066\u3044\u306a\u3044\u7279\u6b8a\u6587\u5b57\u304c\u542b\u307e\u308c\u308b\u5834\u5408\u3002<\/span>
      <\/span><\/li>
    • \u7279\u5b9a\u306e\u5217\u304c\u4e00\u610f\u3067\u3042\u308b\u3053\u3068\uff08\u30d5\u30a1\u30a4\u30eb\u5185\u3068\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u5185\u3067\u91cd\u8907\u304c\u306a\u3044\u3053\u3068\uff09\u3002<\/span>
      <\/span><\/li><\/ul><\/li>
    • \u30e6\u30fc\u30b6\u30fc\u306b\u8fd4\u3059\u60c5\u5831\uff1a<\/span>
      <\/span>
      • \u30a8\u30e9\u30fc\u306e\u3042\u308b\u884c\u3002<\/span>
        <\/span><\/li>
      • \u30a8\u30e9\u30fc\u306e\u7a2e\u985e\u3002<\/span>
        <\/span><\/li>
      • \u554f\u984c\u306e\u3042\u308b\u5024\u3002<\/span><\/li><\/ul><\/li><\/ul>

        \u5168\u4f53\u7684\u306a\u30bd\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3<\/strong><\/h2>

        \u51e6\u7406\u30d5\u30ed\u30fc\u306f\u4ee5\u4e0b\u306e3\u3064\u306e\u30d5\u30a7\u30fc\u30ba\u306b\u5206\u304b\u308c\u3066\u3044\u307e\u3059\uff1a<\/span><\/p>

        1. \u30e6\u30fc\u30b6\u30fc\u304c\u30d5\u30a1\u30a4\u30eb\u3092S3\u306b\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9<\/strong><\/h3>
        • \u30d5\u30ed\u30f3\u30c8\u30a8\u30f3\u30c9\u306f\u3001\u7f72\u540d\u4ed8\u304dURL\u307e\u305f\u306fS3 SDK\u3092\u4f7f\u7528\u3057\u3066\u30d5\u30a1\u30a4\u30eb\u3092S3\u306b\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u3057\u307e\u3059\u3002<\/span>
          <\/span><\/li>
        • \u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u304c\u6210\u529f\u3059\u308b\u3068\u3001EventBridge\u3092\u4ecb\u3057\u3066\u3001\u307e\u305f\u306f\u30d0\u30c3\u30af\u30a8\u30f3\u30c9\u306b\u30ea\u30af\u30a8\u30b9\u30c8\u3092\u9001\u4fe1\u3057\u3066\u691c\u8a3c\u30d7\u30ed\u30bb\u30b9\u3092\u958b\u59cb\u3057\u307e\u3059\u3002<\/span>
          <\/span><\/li><\/ul>

          2. \u30d0\u30c3\u30c1\u51e6\u7406\uff1a\u30c7\u30fc\u30bf\u306e\u691c\u8a3c<\/strong><\/h3>

          \u691c\u8a3c\u30d7\u30ed\u30bb\u30b9\u306f\u3001\u30d1\u30d5\u30a9\u30fc\u30de\u30f3\u30b9\u3092\u6700\u9069\u5316\u3057\u3001\u30bf\u30a4\u30e0\u30a2\u30a6\u30c8\u3084\u30e1\u30e2\u30ea\u30ea\u30fc\u30af\u306e\u30ea\u30b9\u30af\u3092\u6e1b\u3089\u3059\u305f\u3081\u306b\u8907\u6570\u306e\u30b9\u30c6\u30c3\u30d7\u306b\u5206\u3051\u3066\u5b9f\u884c\u3055\u308c\u307e\u3059\u3002<\/span><\/p>

          a. \u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u306b\u4e00\u6642\u30c6\u30fc\u30d6\u30eb\u3092\u4f5c\u6210<\/strong><\/h4>
          • CSV\u69cb\u9020\u3092\u53cd\u6620\u3057\u305f\u4e00\u6642\u30c6\u30fc\u30d6\u30eb\u3092\u4f5c\u6210\u3057\u307e\u3059\u3002<\/span><\/li>
          • \u30a8\u30e9\u30fc\u3092\u30ed\u30b0\u306b\u8a18\u9332\u3059\u308b\u305f\u3081\u306e\u8ffd\u52a0\u5217\u3092\u8ffd\u52a0\uff1a <\/span>column_1_length<\/span>, <\/span>column_1_regex<\/span>, <\/span>is_error<\/span>, <\/span>line_number<\/span>\u306a\u3069\u3002<\/span><\/li><\/ul>

            b. S3\u304b\u3089\u4e00\u6642\u30c6\u30fc\u30d6\u30eb\u306b\u30d5\u30a1\u30a4\u30eb\u3092\u30ed\u30fc\u30c9<\/strong><\/h4>
            • S3\u304c\u30ed\u30fc\u30ab\u30eb\u306b\u30de\u30a6\u30f3\u30c8\u3055\u308c\u3066\u3044\u308b\u5834\u5408\u3001 <\/span>LOAD DATA LOCAL INFILE<\/span> \u3092\u4f7f\u7528\u3057\u3066\u30d5\u30a1\u30a4\u30eb\u3092\u76f4\u63a5\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u306b\u30a4\u30f3\u30dd\u30fc\u30c8\u3057\u307e\u3059\u3002<\/span>
              <\/span><\/li><\/ul>
              • \u30aa\u30d7\u30b7\u30e7\u30f3\u3068\u3057\u3066\u3001\u5024\u306b\u57cb\u3081\u8fbc\u307e\u308c\u305f\u6539\u884c\u6587\u5b57\uff08<\/span>\\r\\n<\/span>\uff09\u3092\u524a\u9664\u3059\u308b\u524d\u51e6\u7406\u3092\u884c\u3044\u307e\u3059\u3002<\/span>
                <\/span><\/li>
              • \u30c7\u30fc\u30bf\u30ed\u30fc\u30c9\u306e\u901f\u5ea6\u3092\u5411\u4e0a\u3055\u305b\u308b\u305f\u3081\u306b\u4e00\u6642\u7684\u306b\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u7121\u52b9\u5316\u3057\u3001\u305d\u306e\u5f8c\u518d\u5ea6\u6709\u52b9\u5316\u3057\u307e\u3059\u3002<\/span><\/li><\/ul>

                LOAD<\/span> DATA<\/span> LOCAL<\/span> INFILE<\/span> ‘\/path_to_file.csv’<\/span>
                <\/span>INTO<\/span> TABLE<\/span> temp_table<\/span>
                <\/span>FIELDS<\/span> TERMINATED<\/span> BY<\/span> ‘,’<\/span>
                <\/span>ENCLOSED<\/span> BY<\/span> ‘”‘<\/span>
                <\/span>LINES<\/span> TERMINATED<\/span> BY<\/span> ‘\\n’<\/span>
                <\/span>(column_1, column_2)<\/span>
                <\/span>SET<\/span>
                <\/span>\u00a0 column_1 = <\/span>REPLACE<\/span>(<\/span>REPLACE<\/span>(column_1, <\/span>‘\\r’<\/span>, <\/span>”<\/span>), <\/span>‘\\n’<\/span>, <\/span>”<\/span>),<\/span>
                <\/span>\u00a0 column_2 = <\/span>REPLACE<\/span>(<\/span>REPLACE<\/span>(column_2, <\/span>‘\\r’<\/span>, <\/span>”<\/span>), <\/span>‘\\n’<\/span>, <\/span>”<\/span>),<\/span>
                <\/span>\u00a0 column_1_length = <\/span>CHAR_LENGTH<\/span>(column_1),<\/span>
                <\/span>\u00a0 column_2_length = <\/span>CHAR_LENGTH<\/span>(column_2),<\/span>
                <\/span>\u00a0 column_1_regex = column_1 REGEXP <\/span>‘[^a-zA-Z0-9+\\-_.:=?&\/]+’<\/span>,<\/span>
                <\/span>\u00a0 is_error = <\/span>IF<\/span> (column_1_length > <\/span>200<\/span> OR<\/span> column_2_length > <\/span>200<\/span> OR<\/span> column_1_regex > <\/span>0<\/span>, <\/span>1<\/span>, <\/span>0<\/span>)<\/span><\/p>

                c. \u30a8\u30e9\u30fc\u30ed\u30a6\u3092\u53d6\u5f97\u3057\u3001\u30a8\u30e9\u30fc\u30ec\u30dd\u30fc\u30c8\u3092\u751f\u6210<\/strong><\/h4>
                • is_error = 1\u306e\u884c\u3092\u9078\u629e\u3057\u3001\u884c\u756a\u53f7\u3068\u30a8\u30e9\u30fc\u306e\u8aac\u660e\u3092\u542b\u3080\u30a8\u30e9\u30fc\u30d5\u30a1\u30a4\u30eb\u306b\u66f8\u304d\u8fbc\u307f\u307e\u3059\u3002<\/li><\/ul>

                  d. \u30d5\u30a1\u30a4\u30eb\u5185\u306e\u91cd\u8907\u3092\u30c1\u30a7\u30c3\u30af<\/strong><\/h4>

                  SELECT<\/span> id_key, <\/span>GROUP_CONCAT<\/span>(line_number) <\/span>AS<\/span> duplicated_lines<\/span>
                  <\/span>FROM<\/span> temp_table<\/span>
                  <\/span>GROUP<\/span> BY<\/span> id_key<\/span>
                  <\/span>HAVING<\/span> COUNT<\/span>(*) > <\/span>1<\/span>;<\/span><\/p>

                  e. \u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u306b\u5bfe\u3059\u308b\u91cd\u8907\u3092\u30c1\u30a7\u30c3\u30af<\/strong><\/h4>

                  temp_table\u306eid_key\u3092\u30b7\u30b9\u30c6\u30e0\u306e\u30e1\u30a4\u30f3\u30c6\u30fc\u30d6\u30eb\u3068\u6bd4\u8f03\u3057\u307e\u3059\u3002<\/span><\/p>

                  3. \u6709\u52b9\u306a\u30c7\u30fc\u30bf\u3092\u30e1\u30a4\u30f3\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u306b\u633f\u5165<\/strong>
                  <\/span><\/h3>
                  • \u30a8\u30e9\u30fc\u306e\u306a\u3044\u884c\u306e\u307f\u3092\u542b\u3081\u307e\u3059\u3002<\/span><\/li>
                  • \u4f7f\u7528\u7528\u9014\u306b\u5fdc\u3058\u3066INSERT INTO\u307e\u305f\u306fREPLACE INTO\u3092\u4f7f\u7528\u3057\u307e\u3059\u3002<\/li>
                  • \u30aa\u30d7\u30b7\u30e7\u30f3\u3068\u3057\u3066\u3001\u30c7\u30d5\u30a9\u30eb\u30c8\u5024\u3001\u76e3\u67fb\u30d5\u30a3\u30fc\u30eb\u30c9\u306a\u3069\u306e\u30ed\u30b8\u30c3\u30af\u3092\u8ffd\u52a0\u3057\u307e\u3059\u3002<\/span>
                    <\/span><\/li><\/ul>

                    4. \u4e00\u6642\u30c6\u30fc\u30d6\u30eb\u306e\u30af\u30ea\u30fc\u30f3\u30a2\u30c3\u30d7<\/strong><\/h3>
                    • \u51e6\u7406\u5f8c\u306b\u4e00\u6642\u30c6\u30fc\u30d6\u30eb\u3092\u524a\u9664\u3057\u3066\u30ea\u30bd\u30fc\u30b9\u3092\u89e3\u653e\u3057\u307e\u3059\u3002<\/span>
                      <\/span><\/li><\/ul>

                      \u30d2\u30f3\u30c8\uff06\u30d9\u30b9\u30c8\u30d7\u30e9\u30af\u30c6\u30a3\u30b9<\/strong><\/h2>
                      • \u5927\u5bb9\u91cf\u30c7\u30fc\u30bf\u30ed\u30fc\u30c9\u4e2d\u306f\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u7121\u52b9\u5316\u3057\u3001\u305d\u306e\u5f8c\u518d\u5ea6\u6709\u52b9\u5316\u3057\u307e\u3059\uff08\u30ed\u30fc\u30c9\u6642\u9593\u3092\u5927\u5e45\u306b\u77ed\u7e2e\uff09\u3002<\/span>
                        <\/span><\/li>
                      • CSV\u304c\u6b63\u3057\u304f\u30a8\u30f3\u30b3\u30fc\u30c9\u3055\u308c\u3066\u3044\u308b\u3053\u3068\u3092\u78ba\u8a8d\u3057\u307e\u3059\uff08UTF-8\u3001BOM\u306a\u3057\uff09\u3002<\/span>
                        <\/span><\/li>
                      • \u30b3\u30fc\u30c9\u3067\u884c\u3054\u3068\u306b\u691c\u8a3c\u3059\u308b\u306e\u3067\u306f\u306a\u304f\u3001\u52b9\u7387\u7684\u306a\u30d0\u30eb\u30af\u691c\u8a3c\u306e\u305f\u3081\u306bSQL\u3092\u4f7f\u7528\u3057\u307e\u3059\u3002<\/span>
                        <\/span><\/li>
                      • \u3059\u3079\u3066\u306e\u30a8\u30e9\u30fc\u3092\u4e00\u5ea6\u306b\u30ed\u30b0\u306b\u8a18\u9332\u3057\u3001\u30e6\u30fc\u30b6\u30fc\u304c\u4e00\u5ea6\u306b\u4fee\u6b63\u3067\u304d\u308b\u3088\u3046\u306b\u3059\u308b\u3002\u9010\u6b21\u4fee\u6b63\u3067\u306f\u306a\u304f\u3001\u4e00\u62ec\u4fee\u6b63\u304c\u53ef\u80fd\u3067\u3059\u3002<\/span>
                        <\/span><\/li><\/ul>

                        \u7d50\u8ad6<\/strong><\/h2>

                        \u5927\u5bb9\u91cf\u30d5\u30a1\u30a4\u30eb\u306e\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u51e6\u7406\u306f\u5358\u306b\u30b5\u30a4\u30ba\u5236\u9650\u3092\u5f15\u304d\u4e0a\u3052\u308b\u3060\u3051\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u30d1\u30d5\u30a9\u30fc\u30de\u30f3\u30b9\u3068\u62e1\u5f35\u6027\u3092\u78ba\u4fdd\u3059\u308b\u305f\u3081\u306b\u3001\u30b9\u30de\u30fc\u30c8\u3067\u52b9\u7387\u7684\u306a\u51e6\u7406\u30d5\u30ed\u30fc\u304c\u5fc5\u8981\u3067\u3059\u3002\u3053\u306e\u30a2\u30d7\u30ed\u30fc\u30c1\u3092\u63a1\u7528\u3059\u308c\u3070\u3001\u5fc5\u8981\u306b\u5fdc\u3058\u3066\u3055\u3089\u306b\u5927\u304d\u306a\u30d5\u30a1\u30a4\u30eb\u306b\u3082\u5bfe\u5fdc\u53ef\u80fd\u3067\u3059\u3002<\/span><\/p>

                        \u540c\u69d8\u306e\u30bd\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3\u3092\u30b3\u30fc\u30c9\u3084\u30af\u30e9\u30a6\u30c9\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306b\u5b9f\u88c5\u3059\u308b\u969b\u306e\u30b5\u30dd\u30fc\u30c8\u304c\u5fc5\u8981\u3067\u3059\u304b\uff1f<\/span><\/p>

                        Credit: Ton That Bach\uff08Mynavi TechTus Vietnam\u306e\u30c6\u30af\u30cb\u30ab\u30eb\u30ea\u30fc\u30c0\u30fc\uff09<\/em><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>","protected":false},"excerpt":{"rendered":"

                        \u73fe\u4ee3\u3067\u306f\u3001\u65c5\u884c\u3092\u542b\u3080\u3042\u3089\u3086\u308b\u6d3b\u52d5\u306b\u304a\u3044\u3066\u3001\u74b0\u5883\u3084\u5730\u57df\u793e\u4f1a\u3078\u306e\u914d\u616e\u3068\u3001\u3088\u308a\u30a8\u30b3\u30d5\u30ec\u30f3\u30c9\u30ea\u30fc\u3067\u3042\u308b\u3053\u3068\u304c\u91cd\u8981\u3067\u3059\u3002<\/p>","protected":false},"author":1,"featured_media":991915,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[75,12],"tags":[107],"class_list":["post-991926","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blogs","category-news","tag-csv"],"acf":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/mynavitechtus.com\/ja\/wp-json\/wp\/v2\/posts\/991926","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mynavitechtus.com\/ja\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mynavitechtus.com\/ja\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mynavitechtus.com\/ja\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mynavitechtus.com\/ja\/wp-json\/wp\/v2\/comments?post=991926"}],"version-history":[{"count":19,"href":"https:\/\/mynavitechtus.com\/ja\/wp-json\/wp\/v2\/posts\/991926\/revisions"}],"predecessor-version":[{"id":992696,"href":"https:\/\/mynavitechtus.com\/ja\/wp-json\/wp\/v2\/posts\/991926\/revisions\/992696"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mynavitechtus.com\/ja\/wp-json\/wp\/v2\/media\/991915"}],"wp:attachment":[{"href":"https:\/\/mynavitechtus.com\/ja\/wp-json\/wp\/v2\/media?parent=991926"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mynavitechtus.com\/ja\/wp-json\/wp\/v2\/categories?post=991926"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mynavitechtus.com\/ja\/wp-json\/wp\/v2\/tags?post=991926"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}