Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
0% found this document useful (0 votes)
1K views

Simplified Way of Processing Large Data Using Chunk in Laravel

Chunking the array or collection is the best way to process large data in PHP. It efficiently manage memory issues and uses less time for script execution.

Uploaded by

Dinesh Suthar
Copyright
© Attribution (BY)
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
1K views

Simplified Way of Processing Large Data Using Chunk in Laravel

Chunking the array or collection is the best way to process large data in PHP. It efficiently manage memory issues and uses less time for script execution.

Uploaded by

Dinesh Suthar
Copyright
© Attribution (BY)
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

Simplified way of

processing large
data using chunk() in
laravel

Learn More

TABLE OF CONTENTS

What do we mean by large data ?

Problems with processing large data ?

What do we mean by data chunks ?

Array chunks in PHP

Best practices to free unused memory in PHP

Unset a php variable

Nullify a php variable

Close database connection in the end manually

https://decodeweb.in
Simplified way of processing large data using chunk() in
laravel
Source: ​https://decodeweb.in/php/simplified-way-of-processing-large-data-using-chunk-in-laravel/

What do we mean by large data ?


As a laravel developer, by large data I mean ​collection of 1000 or more rows​ from a single data model that is,
a database table; specifically on a MySQL / MariaDB server.

Problems with processing large data ?


There may be situations when a single table holds millions of records and you want to migrate those records to
another table after ​processing. ​As far as MySQL is concerned here in this context, PHP has some limitations
in terms of memory allocation and script execution time to process those rows.

PHP takes data into its allocated memory on server and then process further, also it automatically lends some
extra bytes if it thinks it would be sufficient for the script to run and exit successfully. All these actions are
handled automatically until the records are less 1000 but once it crosses this limit server resources become
deficient of memory.

In short, server stops responding to other requests and many script is exited abruptly, which we call it as a
memory leakage. In return, we get a ​Fatal error​: Allowed memory size of <some_number> bytes
exhausted (tried to allocate <some_number> bytes)
On the other hand, when the calculation during data processing is way too much complex then script would
exit abruptly because each script has a definite execution time allocated.

Now you may wonder, that we can manually change these settings in php.ini file but I do not recommend to
mess with any configuration file on the production server unless it is the last resort.

What do we mean by data chunks ?


What if we divide whole collection into chunks, let's say a collection of 25K rows and we divide them in chunks
of 500 rows at a single instance, fair enough ? Yes, because now rows are 500 only at given time. The PHP
script will take only 500 rows out of 25K rows. Let's see some examples related to arrays first.

Array chunks in PHP


In PHP, there is ​an array helper function​, array_chunk().

<?php

$arr_a = [​1​, ​2​, ​3​, ....., ​5000​];

foreach​(array_chunk($arr_a, ​500​) ​as​ $x){


// array_chunk() will divide $arr_a into smaller array as [[1, 2, 3...,
500],[501, 502, .... , 1000] and so one till 5000]

//do some stuff with $x;


}

Laravel Eloquent chunks


In ​Laravel​, collection can be chunked using chunk() method for example,

<?php

Namespace​ ​App​\​Http​\​Controller​;
Use​ ​User​;

Public​ ​function​ ​processUsers​()


{
User::chunk(​500​, ​function​($chunked_collection)​{
//do some stuff with $chunked_collection
});
}
Best practices to free unused memory in PHP
PHP provides many ways to flush unused memory so that the next line of code can be accommodated into
available memory.

Unset a php variable


Using unset() method, we can free up the memory a variable is holding up, for example,

<?php

$a = ​500​;

unset​($a);

var_dump($a); ​// prints null

?>

Nullify a php variable


Another way to free up memory is to reset the variable’s value to null explicitly, but it is slower than unset(), I
do not recommend it.

Close database connection in the end manually


While using core PHP, always make sure to ​close database connection​ after processing has been done on
data. For example:

<?php
$con=mysqli_connect(​"localhost"​,​"my_user"​,​"my_password"​,​"my_db"​);

// ....some PHP code...

mysqli_close($con);
?>

You might also like