Gearman jobqueue

Gearman Job Queue.
Раскрываем недокументированные возможности

Синхронное выполнение задач

Client Client Client Client Client Client Client

Task

Без очередей — хаос!

Очереди к ограниченным ресурсам необходимы!

● Пред-расчет данных
● Операции импорта/экспорта данных
● Обработка логов системы
● Маркетинговый анализ базы данных
● Почтовые рассылки
● Map/Reduce
● и многое другое...

Выполнение задачи в асинхронном режиме
Client Job Server Worker

add job
job handle
execute

success

Client Client Client
Client Client Client
Client

Task manager

Worker Worker Worker Worker Worker

About project Gearman

Gearman = Manager

2005: originally Perl implementation by Danga Interactive
2008: rewrite in C by Brian Aker

Used in:
Digg (45+ servers, 400K jobs/day)
Yahoo (60+ servers, 6M jobs/day)
Xing.com
Grooveshark
...

Преимущества и недостатки

● Высокая скорость ● Мониторинг, администрирование
● Масштабируемость ● Слабая документация
● Балансировка нагрузки ● Работа с памятью
● Параллельное и асинхронное ● Отличия API в различных версиях
выполнение задач
● Защита от потери очереди
● Multi-language API

Gearman server monitoring

Administrative Protocol
> telnet localhost 4730
> Trying ::1...
> Connected to localhost.
> Escape character is '^]'.
> wokers
FD IP-ADDRESS CLIENT-ID : FUNCTION
> status
FUNCTION TOTAL RUNNING AVAILABLE_WORKERS

GearUp: Gearman based monitoring service
https://launchpad.net/gearup

Mod Gearman
http://labs.consol.de/nagios/mod-gearman/

Client Client Client Client Client Client

Client API Client API

Gearman Gearman

Worker API Worker API

Worker Worker Worker Worker Worker

client.php

<?php
# create our object
$client= new GearmanClient();

# add the default server
$client->addServer();

$workload = array(
'email' => 'nobody@example.com' ,
'subject' => 'the subject' ,
'message' => 'the message'
);

# send email
$client->doBackground( "send_mail" , json_encode($workload));

if ($client->returnCode() != GEARMAN_SUCCESS) {
echo "bad return coden" ;
exit;
}
?>

worker.php

<?php
# create the worker
$worker = new GearmanWorker();

# add the default job server (localhost)
$worker->addServer();

# add the reverse function
$worker->addFunction( "send_mail" , "sendMail" );

# start to worker listening for job submissions
while ($worker->work());

function sendMail($job)
{
$arg = json_decode($job->workload(), true);
mail($arg[ 'email'], $arg[ 'subject' ], $arg[ 'message' ]);
echo "email has been sentn" ;
}
?>

Client Gearman Worker Task

register tasks
work()

doBackground()
run task execute

success

worker.php
<?php
#...

{
$arg = json_decode($job->workload(), true);

# connect to remote service
$mail = new Mail($arg['email'], $arg['subject'], $arg['message']);

# send email
if (!$mail->send()){

# it needs to be solved

} else {
echo "email has been sentn";
}
}
?>


doBackground()
run task execute

failure

???

worker.php
<?php
#...

{
#...

# send email

sleep(20);
$client->doBackground("send_mail", $job->workload());
echo "error has happened n";

} else {
}
}
?>

doBackground() run task execute

sleep(20)
doBackground()
failure

run task execute

sleep(20)
doBackground()
failure

run task execute

success

worker.php

<?php
#...

{
#...

# send email

setCron("send_mail", $job->workload(), 20);
echo "error has happened n";

} else {
}
}
?>

Client Gearman Worker Task Cron

failure do postponed

delay
doBackground()

run task execute
do postponed
failure

delay
doBackground()

run task execute

success do postponed

Solution

Credit:

— In ver. 0.20 added when_to_run tasks functionality
— But public API doesn't support that functionality
— Neither does PECL Gearman module
— Documentation also is silent about it

John Ewart
Software Engineer

worker.php
<?php
#...

{
#...

# send email

$time = time() + 20;
$client->doSuspended("send_mail", $job->workload(), NULL, $time);
echo "error has happened n" ;

} else {
echo "email has been sentn" ;
}
}
?>

Suspended task execution
doSuspended()
failure

delay

run task execute
doSuspended()
failure

delay

run task execute

success

client.php

<?php

$workload = array('value' => 1, 'limit_attempts' => 3);

$client->doBackground( "remote_procedure_call", json_encode($workload));

if ($client->returnCode() != GEARMAN_SUCCESS) {
echo "bad return coden" ;
exit;
}
?>

worker.php
<?php
#...
$worker->addFunction( 'remote_procedure_call' , 'rpcJobHandler' );
while ($worker->work());

function rpcJobHandler($job)
{
$args = json_decode($job->workload(), true);
try { // Execute task
remoteProcedureCall($args[ 'value']);
} catch (Exception $e) {
echo "Failure, reason: " . $e->getMessage() . "n";
}
}

function remoteProcedureCall(&$value)
{ // Accept value only less than 3
if ($value < 3) {
$value++;
throw new Exception ("Service is unavailable" );
}
}
?>

worker.php — break of task loop
<?php
#...
function rpcJobHandler($job)
{
$args = json_decode($job->workload(), true);
try {
remoteProcedureCall($args[ 'value']);
} catch (Exception $e) {
if (!isset($args['attempt' ])) $args[ 'attempt' ] = 0;
if (empty($args['limit_attempts' ]) or $args['attempt' ] <= $args[ 'limit_attempts' ]) {
$client = new GearmanClient();
if (method_exists($client, 'doSuspended' )) {
$time = getExecutionTime($args['attempt' ]);
$args[ 'attempt' ]++;
$client->doSuspended('run', json_encode($args), null, $time);
}
} else {
echo "Failure, reason: " . $e->getMessage() . "n";
}
}
}
?>

Geometric progression
<?php
#...
function getExecutionTime($attempt)
{
return time() + pow(2, (int)$attempt) * 10;
}
?>

attempt seconds delta time
0 10 10s
1 20 20s
2 40 40s
3 80 1m 20s
4 160 2m 40s
5 320 5m 20s
6 640 10m 40s
7 1280 21m 20s
8 2560 42m 40s
9 5120 1h 25m 20s
10 10240 2h 50m 40s

Gearman jobqueue

More Related Content

Gearman jobqueue