AWS WLM CLI

WLM Dynamic Memory Allocation

 In each queue, WLM creates a number of query slots equal to the queue's
concurrency level.

 The amount of memory allocated to a query slot equals the percentage of

memory allocated to the queue divided by the slot count.

 If we change the memory allocation or concurrency, Amazon Redshift dynamically

manages the transition to the new WLM configuration so that active queries can
run to completion using the currently allocated amount of memory
 The WLM uses the following process to manage the transition.

1. WLM recalculates the memory allocation for each new query slot.

2. If a query slot is not actively being used by a running query, WLM removes the
slot, which makes that memory available for new slots.

3. If a query slot is actively in use, WLM waits for the query to finish.

4. As active queries complete, the empty slots are removed and the associated
memory is freed.

5. As enough memory becomes available to add one or more slots, new slots are
added.
6. When all queries that were running at the time of the change finish, the slot
count equals the new concurrency level, and the transition to the new WLM
configuration is complete.
Example : http://docs.aws.amazon.com/redshift/latest/dg/cm-c-wlm-dynamic-
example.html
Note
• Queries that are running when the change takes place continue to use the original
memory allocation and queries that are queued when the change takes place are
routed to new slots as they become available.

• To view the status of the transition, query the STV_WLM_SERVICE_CLASS_CONFIG

system table.

• Aws CLI command should be in one single line.

WLM Implementation using Command Line Interface
To configure WLM, we modify the “wlm_json_configuration” parameter.
Syntax:
In the below json configuration we have created 2 queue and each queue having 2 properties.
[
{
"ParameterName":"wlm_json_configuration", "ParameterValue":
"[
{
"q1_first_property_name":"q1_first_property_value",
"q1_second_property_name":"q1_second_property_value",
...
},
{
"q2_first_property_name":"q2_first_property_value",
"q2_second_property_name":"q2_second_property_value",
...
}

]"
}
]
 Rules for Configuring WLM by Using the AWS CLI in the Command Line on
the Linux and Mac OS X Operating Systems
1. The entire JSON structure must be enclosed in single quotation marks (') and
brackets ([ ]).
2. All parameter names and parameter values must be enclosed in double quotation
marks (").
3. Within the ParameterValue value, you must enclose the entire nested structure in
double-quotation marks (") and brackets ([ ]).
4. Within the nested structure, each of the properties and values for each queue
must be enclosed in curly braces ({ }).
5. Within the nested structure, you must use the backslash (\) escape character
before each double-quotation mark (").
6. For name/value pairs, a colon (:) separates each property from its value.
7. Each name/value pair is separated from another by a comma (,).
8. Multiple queues are separated by a comma (,) between the end of one queue's
curly brace (}) and the beginning of the next queue's curly brace ({).
 AWS CLI Command
Without Rule

aws redshift modify-cluster-parameter-group --parameter-group-name test-wlm

--parameters
'[{"ParameterName":"wlm_json_configuration","ParameterValue":"[ { \"query_gr
oup\" : [ \"small_query\" ], \"user_group\" :
[ \"admin_test\" ], \"query_group_wild_card\" : 0, \"user_group_wild_card\" : 0,
\"query_concurrency\" : 3, \"max_execution_time\" :
0, \"memory_percent_to_use\" : 30},{ \"query_group\" :
[ \"big_query\" ], \"user_group\" : [ \"dev_test\" ], \"query_group_wild_card\" :
0, \"user_group_wild_card\" : 0, \"query_concurrency\" :
4, \"max_execution_time\" : 0, \"memory_percent_to_use\" : 25} ,
{ \"query_group\" : [ ], \"user_group\" : [ ], \"query_group_wild_card\" :
0, \"user_group_wild_card\" : 0, \"query_concurrency\" :
5, \"max_execution_time\" : 0, \"memory_percent_to_use\" :
40} ]","ApplyType":"dynamic"}]'
 AWS CLI Command
With Rule and SQA

aws redshift modify-cluster-parameter-group --parameter-group-name test-wlm

--parameters
'[{"ParameterName":"wlm_json_configuration","ParameterValue":"[ {\"query_gro
up\" : [ \"mni_query_grp\" ], \"user_group\" :
[ \"mni_user_grp\" ], \"query_group_wild_card\" : 0, \"user_group_wild_card\" :
0, \"query_concurrency\" : 1, \"max_execution_time\" :
20000, \"memory_percent_to_use\" : 15,\"rules\" :
[ { \"rule_name\": \"rule_2\", \"predicate\" :
[ { \"metric_name\": \"scan_row_count\", \"operator\": \">\", \"value\":
10000} ], \"action\": \"hop\"} ]},{ \"query_group\" : [ ], \"user_group\" :
[ ], \"query_group_wild_card\" : 0, \"user_group_wild_card\" :
0, \"query_concurrency\" : 4, \"max_execution_time\" :
0, \"memory_percent_to_use\" : 25},{ \"short_query_queue\":
true, \"max_execution_time\": 5000}]","ApplyType":"dynamic"}]'
 Prompt message after execution:

ParameterGroupStatus": "Your parameter group has been

updated. Updates to dynamic parameters will be applied
immediately. Updates to static parameters will be applied when
you reboot the associated Clusters.", "ParameterGroupName":
"test-wlm“.
 Test Case

Dynamic Property changes: concurrency and memory changes.

aws redshift modify-cluster-parameter-group --parameter-group-name test-wlm

--parameters
'[{"ParameterName":"wlm_json_configuration","ParameterValue":"[ { \"query_group\"
: [ \"small_query\" ], \"user_group\" : [ \"admin_test\" ], \"query_group_wild_card\" :
0, \"user_group_wild_card\" : 0, \"query_concurrency\" :
2, \"max_execution_time\" : 0, \"memory_percent_to_use\" : 20},
{ \"query_group\" : [ ], \"user_group\" : [ ], \"query_group_wild_card\" :
0, \"user_group_wild_card\" : 0, \"query_concurrency\" :
5, \"max_execution_time\" : 0, \"memory_percent_to_use\" :
50} ]","ApplyType":"dynamic"}]‘

Conclusion : It took approximately 7 minutes to come from modifying to available

status. No Reboot required. We can run the query during the modifying status and
once the new slots will be available it will start executing the query.
 Cluster reboot using CLI

Syntax for reboot:(Incase static property changes)

aws redshift reboot-cluster --cluster-identifier cluster_name

Syntax to check cluster status:

aws emr describe-cluster --cluster-id j-XXXXXXXX

where j-XXXXXXXX is cluster-id which we want to see the status of

cluster.