"
Supported by

Teach ABAP to juggle

What can be done when a night is not enough to complete the daily processes?


Why?

In the course of time, the volume of daily jobs executed in the SAP systems that are processed during the night, keeps on increasing. However, the concept of night, that is, the time with less active users in the system, becomes more and more meaningless when considering a multi-country system where the users are distributed all over the world. Those jobs consist mostly of integration/recalculation processes, which may involve a substantial amount of data.

Theses types of processes are normally independent, i.e., they are not dependent on the execution of a particular item before processing the next. Taking this assumption into account, it would be extremely useful to be able to process the items parallelly. This separation can be achieved by implementing an infrastructure that allows for parallel processing.


What is parallel processing?

Parallel processing, as the name refers, is nothing else than processing simultaneously those tasks, that are independent from each other. It is a similar concept as the threads in some of the OOP languages (C#, C++, Java).
In practice, parallel processing is the execution of asynchronous RFC, using an RFC server group. The RFC server group is an SAP concept that allows for configuring an identifier and associate multiple application servers as well as the percentage of available resources.

How does it work?


CALL FUNCTION func STARTING NEW TASK task 
              [DESTINATION {dest|{IN GROUP {group|DEFAULT}}}] 
              [{PERFORMING subr}|{CALLING meth} ON END OF TASK] 
              parameter_list.

The smallest constituent of this approach is an RFC enabled function module, aimed at processing one single item (or a block of items, but let’s keep it simple for now). And the connecting point between an RFC and the RFC server group is a keyword IN GROUP <name defined in RZ12>.

You can also define a return callback, that is executed when this process is finished, keyword PERFORMING subr ON END OF TASK or CALLING method ON END OF TASK.

Comparing parallel and sequential processing

Let’s have a look at an example that compares two programs which have the same objective. Let’s assume the following objective:

“For each country configured in the system in T005 table we want to run a specific calculation and store the result in a Z-table.”

To achieve this, we use the “Z_PP_UNIT_TESTS” function module, that will be invoked by both programs.


FUNCTION Z_PP_UNIT_TESTS.
*"----------------------------------------------------------------------
*"*"Interface local:
*"  IMPORTING
*"     VALUE(DATA) TYPE  T005 OPTIONAL
*"  EXPORTING
*"     VALUE(RETURNINFO) TYPE  ZPP_EXECUTIONINFO
*"----------------------------------------------------------------------
  data: random_num  type qfranint,
        l_odd       type int4,
        l_count     type int4.

  returninfo-server = sy-host.
  CALL FUNCTION 'TH_GET_OWN_WP_NO'
    IMPORTING
      wp_pid = returninfo-wpinfo-wp_pid.

  WHILE 1 = 1.

    CALL FUNCTION 'QF05_RANDOM_INTEGER'
      EXPORTING
        ran_int_max = 100
        ran_int_min = 10
      IMPORTING
        ran_int     = random_num.

    l_odd = random_num mod 2.

    IF l_odd = 0.
      add 1 to l_count.
    ENDIF.

    IF data-XADDR = 'X' and l_count > 100000.
      exit.
    ELSEIF l_count > 150000.
      exit.
    ENDIF.

Sequential Processing

Due to the length and complexity of the code, we provide it separately in a file, that can be checked here

Here I show you the main part, the loop:


  loop at it_t005 into wa_005.
    clear l_returninfo.

    CALL FUNCTION 'Z_PP_UNIT_TESTS'
      EXPORTING
        DATA       = wa_005
      IMPORTING
        RETURNINFO = l_returninfo.
    DO.
      ASSIGN COMPONENT sy-index OF STRUCTURE wa_005 TO .
      IF sy-subrc NE 0.
        EXIT.
      ENDIF.
      move  TO tp_written.
      CONCATENATE tp_string_concat tp_written INTO tp_string_concat.
    ENDDO.
    wa_proc-info = tp_string_concat.
    wa_proc-server = l_returninfo-server.
    append wa_proc to git_processed.
  ENDLOOP.

Measurement:

processamento_sequencial_analise

Trace:

processamento_sequencial_trace

Parallel processing

The example program for parallel processing can be checked here.

I show here only the main parts, the loop and the callback routine:


  loop at it_t005_data into wa_005_data.
    l_tabix = sy-tabix.
    CONCATENATE 'Task' l_tabix into taskname.
    clear l_returninfo1.

    CALL FUNCTION 'Z_PP_UNIT_TESTS'
      STARTING NEW TASK taskname
      DESTINATION in group groupname
      PERFORMING callback_meth ON END OF TASK
      EXPORTING
        DATA                  = wa_005_data
      EXCEPTIONS
        resource_failure      = 1
        system_failure        = 2
        COMMUNICATION_FAILURE = 3.
    IF sy-subrc = 0.
      add 1 to gv_send.
      DO.
        ASSIGN COMPONENT sy-index OF STRUCTURE wa_005_data TO .
        IF sy-subrc NE 0.
          EXIT.
        ENDIF.
        WRITE  TO tp_written1.
        CONCATENATE tp_string_concat1 tp_written1 INTO tp_string_concat1.
      ENDDO.
      wa_proc1-info = tp_string_concat1.
      append wa_proc1 to git_processed1.
    else.
      " Erro no processamento, efectuar nova tentativa? registar em log? ...
      " esperar pela libertação de recursos?
      " ...
    ENDIF.

  ENDLOOP.

  WAIT UNTIL gv_send >= gv_receive.
  " parallel processing is now finalized

----------------

FORM callback_meth using taskname.
  data: l_retinfo type ZPP_EXECUTIONINFO.

  RECEIVE RESULTS FROM FUNCTION 'Z_PP_UNIT_TESTS'
  IMPORTING
    returninfo = l_retinfo
    EXCEPTIONS
        resource_failure      = 1
        system_failure        = 2
        COMMUNICATION_FAILURE = 3.
  IF sy-subrc <> 0.
    " controlo de erros ... retry? log?
    " ...
  ENDIF.
  add 1 to gv_receive.

ENDFORM.                    "callback_meth

Measurement:

processamento_paralelo_analise

Traces (there are several because there are various processes running):

processamento_paralelo_trace4

processamento_paralelo_trace3

processamento_paralelo_trace2

processamento_paralelo_trace1

I’m convinced. How can I use it?

In order to use this tool we have to consider some aspects, such as:

  • Invoke the RFC with the respective RFC Server Logon Group;
  • Currently not enough resources in the RFC Server Logon Group available;
  • Need to encode a callback by coding

To account for these and other aspects we always need to take back some coding, that is, use the old technique of copy, paste and change some stuff.

To make things easy, I created a framework that allows for simplifying this process by grouping all common logic, making it easy and practical to perform changes centrally. Here’s an example of how to use it:


report  zrep_pp_unit_test.

include zrep_pp_unit_test_top.
include zrep_pp_unit_test_dat.
include zrep_pp_unit_test_frm.

initialization.
  clear sp_loggr.
  sp_maxt = 3600.
  sp_tswt = 2.
  sp_maxr = 5.
  sp_tstn = 1.
  sp_logl = 0.

start-of-selection.

  " ****************************************************************
  " Retrieve some example data
  data: it_t005 type table of t005,
        l_input type int4.
  l_input = sp_lins.
  IF l_input > 0 and l_input < 255.
    " do nothing on purpose
  else.
    l_input = 10.
  ENDIF.

  select * into table it_t005 up to l_input rows from t005.

  try .
      " ************************************************************
      " Build instance
      l_pp_ref = zcl_pp_factory=>build_instance( 
        pit_raw_data            = it_t005
        p_rfc_name              = 'Z_PP_UNIT_TESTS'
        p_logon_group           = sp_loggr
        p_max_execution_time    = sp_maxt
        p_task_wait_time        = sp_tswt
        p_task_max_retries      = sp_maxr
        p_task_wait_no_resource = sp_tstn
        p_log_level             = sp_logl ).

      " ************************************************************
      " Execute the processing
      l_pp_ref->run( ).

      " ************************************************************
      " Display results
      perform display_result using
                                l_pp_ref
                              changing
                                git_processed
                                git_unprocessed
                                git_error.

    catch zcx_pp_exception into cx.
      l_err = cx->if_message~get_text( ).
      message l_err type 'S' DISPLAY LIKE 'E'.
  endtry.

Where can I get this framework?

This framework was created to simplify and reduce the complexity inherent to the implementation of a process that uses parallel processing (creation of the method with a specific signature for callback, code for controlling repetitions and reprocessing, which was constantly copied and pasted, more complex code maintenance, among many other associated problems).

Get here the associated documentation for this Framework, as well as all the source code provided in SAPLINK format.

Greetings from António Vaz.

From Abapinho, many thanks and greetings to António Vaz.

2 comentários a “Teach ABAP to juggle”

  1. José Vília Diz:

    Bom dia,

    Já utilizo este tipo de processamento de há dois anos para cá e acho que é uma das melhores ferramentas que temos para poder paralelizar processos grandes alem de que as primeiras vezes que o montamos é desafiante! :)

    Só ha um ponto que gostaria de partilhar:

    No meu ver, este tipo de processamento não pode ocupar todos os recursos que temos disponíveis no server group.

    Dando um exemplo prático:
    Se vou implementar um processo paralelo, e antes do meu processo começar, na cadeia batch, temos outro idêntico que não restringe o numero de processos que lança, consumindo todos os recursos do server group, bem que posso fazer código corrido porque sempre que chamar a minha função sairá sempre com a excepção resource_failure e terei que a chamar online.

    Para controlar o número de processos que chamamos, habitualmente utilizo duas funções que me dão a informação que preciso. São as funções: SPBT_INITIALIZE e SPBT_GET_CURR_RESOURCE_INFO e chamo-as assim:

    IF p_paral EQ ‘X’.
    * Se foi marcada opção de processamento paralelo no ecrã de seleção..

    * Inicializamos o server group e obtemos os processos máximos e os disponíveis:
    CALL FUNCTION ‘SPBT_INITIALIZE’
    EXPORTING
    group_name = lv_group_name “nome do server group da RZ12
    IMPORTING
    max_pbt_wps = lv_max_pbt_wps
    free_pbt_wps = lv_free_pbt_wps
    EXCEPTIONS
    invalid_group_name = 1
    internal_error = 2
    pbt_env_already_initialized = 3
    currently_no_resources_avail = 4
    no_pbt_resources_found = 5
    cant_init_different_pbt_groups = 6
    OTHERS = 7.

    IF sy-subrc = 3.
    * Como já foi inicializado, apenas obtemos os recursos do grupo de funções:
    CALL FUNCTION ‘SPBT_GET_CURR_RESOURCE_INFO’
    IMPORTING
    max_pbt_wps = lv_max_pbt_wps
    free_pbt_wps = lv_free_pbt_wps
    EXCEPTIONS
    internal_error = 1
    pbt_env_not_initialized_yet = 2
    OTHERS = 3.
    ENDIF.

    IF sy-subrc EQ 0.

    * Neste perform, apenas faço uma regra:
    * Se os processos máximos são mais que 30 (um valor que me lembrei :) ) e temos pelo menos 60% disponíveis
    * Utilizo 4 processos paralelos, caso contrario, chamo todas as sequencias de threads online…

    PERFORM check_number_of_processes USING lv_max_pbt_wps
    lv_free_pbt_wps
    CHANGING lv_tasks_to_use.

    * Estas 4 tarefas, no meu caso eram suficientes, os valores que menciono não têm qualquer tipo de
    * critério adicional, a não ser o bom senso VS a necessidade do processo! :)
    ENDIF.

    ENDIF.

    * Qualquer outra excepção que ocorra, executo o processamento online pois deverá haver alguma indisponibilidade…

  2. Alfredo Araujo Diz:

    Antônio,

    Estou tentando replicar esse código, as vc não disponibilizou essa função Z_PP_UNIT_TESTS
    Poderia me enviar esses códigos para estudar ?

    Obrigado

Deixe um comentário


About Abapinho
Abapinho runs on WordPress
Articles (RSS) e Comments (RSS).