{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Policy learning and Policy Gradients\n", "\n", "This is a recap of policy learning contextualizes how we can use MJX's differentiability for policy learning. If the below concepts are unfamiliar, there are many great resources [online](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html)! \n", "\n", "The goal of policy learning is to find a control policy $\\pi$ which outputs actions $a_t \\sim \\pi(\\cdot| x_t, \\theta)$ maximizing the total rewards $\\sum r_t$ over some time period, where $r_t$ is shorthand for a reward function evaluated at the state and action of time t:\n", "\n", "$$r_t = r(x_t, a_t)$$\n", "\n", "$\\theta$ are the parameters of the policy; weights in the common case that the policy is a neural network. **Policy gradient methods** involve estimating the gradient of the rewards with respect to the weights, and using this value in a first-order optimization algorithm such as Gradient Descent or [Adam](https://arxiv.org/abs/1412.6980). How we estimate the policy gradient depends on what state transition model we assume. \n", "\n", "#### Zeroth-Order Policy Gradients (ZoPG)\n", "\n", "Referring to `mjx.step` as the simulation function f, we borrow some [terminology](https://arxiv.org/abs/2202.00817) to differentiate between zeroth-order gradients, which only depend on values of f, and first-order gradients, which depend on its jacobian.\n", "\n", "Reinforcement learning (RL) algorithms such as the standard [PPO](https://github.com/google/brax/blob/main/brax/training/agents/ppo/train.py) assume the stochastic state transition model $x_{t+1} \\sim P(\\cdot | x_t, a_t)$. This leads to a ZoPG of the form:\n", "\n", "$$\n", "\\nabla_\\theta J(\\pi_\\theta) = \\mathbb{E}_{\\tau \\sim \\pi_\\theta}\\left[ \\sum \\nabla_\\theta \\log\\pi_\\theta (a_t | s_t) R(\\tau) \\right]\n", "$$\n", "\n", "Where $R(\\tau)$ is some function depending on the rollout $\\tau = \\{x_t, a_t\\}_{t=0}^{T}$. Despite this method's popularity and extensive research into its refinement, a fundamental property is that the gradient has high variance. This allows the optimizer to thoroughly explore the space of policies, leading to the robust and often surprisingly good policies that have been achieved. However, the variance comes at the cost of requiring many samples $(x_t, a_t)$ to converge.\n", "\n", "#### First-Order Policy Gradients (FoPG)\n", "On the other hand, if you assume a deterministic state transition model $x_{t+1} = f(x_t, a_t)$, you end up with the first-order policy gradient. Other common names include Analytical Policy Gradients (APG) and Backpropogation through Time (BPTT). Unlike ZoPG methods, which model the state evolution as a probabilistic black box, the FoPG explicitly contains the jacobians of the simulation function f. For example, let's look at the gradient of the reward $r_t$, in the case that it only depends on state.\n", "$$\n", "\\frac{\\partial r_t}{\\partial \\theta} = \\frac{\\partial r_t}{\\partial x_t}\\frac{\\partial x_t}{\\partial \\theta} \n", "$$\n", "\n", "$$\n", "\\frac{\\partial x_t}{\\partial \\theta} = \\textcolor{Navy}{\\frac{\\partial f(x_{t-1}, a_{t-1})}{\\partial x_{t-1}}}\\frac{\\partial x_{t-1}}{\\partial \\theta} + \\textcolor{Navy}{\\frac{\\partial f(x_{t-1}, a_{t-1})}{\\partial a_{t-1}}} \\frac{\\partial a_{t-1}}{\\partial \\theta}\n", "$$\n", "\n", "The navy-colored terms in the above expression are enabled by MJX's differentiability and are the key difference between FoPG's and ZoPG's. An important consideration is what these jacobians look like near contact points. To see why certain gradients within the jacobian can be pathological, imagine a hard sphere falling toward a block of marble. How does its velocity change with respect to distance ($\\frac{\\partial \\dot{z}_t}{\\partial z_t}$), the instant before it touches the ground? This is the case of an **uninformative gradient**, due to [hard contact](https://arxiv.org/html/2404.02887v1). Fortunately, the default contact settings in Mujoco are sufficiently [soft](https://mujoco.readthedocs.io/en/stable/computation/index.html#soft-contact-model) for learning via FoPG's. With soft contacts, the ground applies an increasing force on the ball as it penetrates it, unlike rigid contacts, which instantly provide enough force for deflection.\n", "\n", "A helpful way to think about FoPG's is via the chain rule and computation graphs, as illustrated below for how $r_2$ influences the policy gradient, again for the case that the reward does not depend on action:\n", "\n", "\"drawing\"\n", "\n", "Note that there three distinct gradient chains in this example. The red pathway considers how the immediately prior action affected the state. The blue path explains the name *Backpropogation through Time*, capturing how actions affect downstream rewards. The least intuitive may be the green chain, which shows how the reward depends on how actions depend on previous actions.Experience shows that blocking *any* of these three pathways via jax.lax.stop_grad can badly hinder policy learning. As the length of $x_t$ backbone increases, [gradient explosion](https://arxiv.org/abs/2111.05803) becomes a crucial consideration. In practice, this can be resolved via decaying downstream gradients or periodically truncating the gradient.\n", "\n", "**The Sharp Bits of FoPG's**\n", "\n", "While FoPG's have been shown to be very sample efficient, especially as the [dimension of the state space increases](https://arxiv.org/abs/2204.07137), one fundamental shortcoming is that due to the lower gradient variance, FoPG's also have less exploration power than ZoPG's and benefit from the practioner being more explicit in the problem formulation.\n", "\n", "Additionally, discontinuous reward formulations are ubiquitious in RL, for instance, a large penalty when the robot falls. It can be significantly more [challenging](https://arxiv.org/abs/2403.14864) to design robust policies with FoPG's, since they cannot backprop through such penalties.\n", "\n", "Last, despite the sample efficiency, FoPG methods can still struggle with wall-clock time. Because the gradients have low variance, they do not benefit significantly from massive parallelization of data collection - unlike [RL](https://arxiv.org/abs/2109.11978). Additionally, the policy gradient is typically calculated via autodifferentiation. This can be 3-5x slower than unrolling the simulation forward, and memory intensive, with memory requirements scaling with $O(m \\cdot (m+n) \\cdot T)$, where m and n are the state and control dimensions, $m \\cdot (m+n)$ is the jacobian dimension, and T is the number of steps propogated through.\n", "\n", "Note that with certain models, using autodifferentiation through mjx.step currently causes [nan gradients](https://github.com/google-deepmind/mujoco/issues/1517). For now, we address this issue by using double-precision floats, at the cost of doubling the memory requirements and training time." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Coming Up**\n", "\n", "In this tutorial, we demonstrate two ways to use FoPG's, using Brax's simple APG [algorithm](https://github.com/google/brax/tree/main/brax/training/agents/apg). This algorithm uses FoPG's to essentially perform live stochastic gradient descent on the policy, unrolling it for a short window, doing a policy update, then continuing where it left off." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup: Imports and installations" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"0.8\" # 0.9 causes too much lag. \n", "from datetime import datetime\n", "import functools\n", "\n", "# Math\n", "import jax.numpy as jp\n", "import numpy as np\n", "import jax\n", "from jax import config # Analytical gradients work much better with double precision.\n", "config.update(\"jax_debug_nans\", True)\n", "config.update(\"jax_enable_x64\", True)\n", "config.update('jax_default_matmul_precision', jax.lax.Precision.HIGH)\n", "import matplotlib.pyplot as plt\n", "\n", "# Sim\n", "import mujoco\n", "import mujoco.mjx as mjx\n", "\n", "# Brax\n", "from brax import envs\n", "from brax import envs\n", "from brax import math\n", "from brax.base import Motion, Transform\n", "from brax.io import mjcf\n", "from brax.envs.base import PipelineEnv, State\n", "from brax.mjx.pipeline import _reformat_contact\n", "from brax.training.acme import running_statistics\n", "from brax.io import model\n", "\n", "# Algorithms\n", "from brax.training.agents.apg import train as apg\n", "from brax.training.agents.apg import networks as apg_networks\n", "from brax.training.agents.ppo import train as ppo\n", "\n", "# Supporting\n", "from etils import epath\n", "import mediapy as media\n", "from ml_collections import config_dict\n", "from typing import Any, Dict\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Quadruped Env" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "!git clone https://github.com/google-deepmind/mujoco_menagerie.git" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "xml_path = epath.Path('mujoco_menagerie/anybotics_anymal_c/scene_mjx.xml').as_posix()\n", "\n", "mj_model = mujoco.MjModel.from_xml_path(xml_path)\n", "\n", "if 'renderer' not in dir():\n", " renderer = mujoco.Renderer(mj_model)\n", "\n", "init_q = mj_model.keyframe('standing').qpos\n", "\n", "mj_data = mujoco.MjData(mj_model)\n", "mj_data.qpos = init_q\n", "mujoco.mj_forward(mj_model, mj_data)\n", "\n", "renderer.update_scene(mj_data)\n", "media.show_image(renderer.render())" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# Rendering Rollouts\n", "def render_rollout(reset_fn, step_fn, \n", " inference_fn, env, \n", " n_steps = 200, camera=None,\n", " seed=0):\n", " rng = jax.random.key(seed)\n", " render_every = 3\n", " state = reset_fn(rng)\n", " rollout = [state.pipeline_state]\n", "\n", " for i in range(n_steps):\n", " act_rng, rng = jax.random.split(rng)\n", " ctrl, _ = inference_fn(state.obs, act_rng)\n", " state = step_fn(state, ctrl)\n", " if i % render_every == 0:\n", " rollout.append(state.pipeline_state)\n", "\n", " media.show_video(env.render(rollout, camera=camera), \n", " fps=1.0 / (env.dt*render_every),\n", " codec='gif')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Study 1: Imitating Kinematics\n", "\n", "FoPG's have been demonstrated to work well in [imitation learning](https://openreview.net/forum?id=06mk-epSwZ), especially when the agent state is reset to track the reference state when it gets too far. In this section, we learn to trot in place. Due to the limited gain on the PD controllers, trotting quickly is far from trivial!\n", "\n", "The RL environment has three rewards:\n", "- min_reference_tracking penalizes error from the reference motion, in minimal coordinates. This makes the policy's output more precise.\n", "- reference_tracking penalizes error in maximal coordinates and increases training stability.\n", "- feet_height nudges the balance of which body positions and velocities to track, by placing additional incentive on the *position* of the feet." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Designing Reference Kinematics" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "def cos_wave(t, step_period, scale):\n", " _cos_wave = -jp.cos(((2*jp.pi)/step_period)*t)\n", " return _cos_wave * (scale/2) + (scale/2)\n", "\n", "def dcos_wave(t, step_period, scale):\n", " \"\"\" \n", " Derivative of the cos wave, for reference velocity\n", " \"\"\"\n", " return ((scale*jp.pi) / step_period) * jp.sin(((2*jp.pi)/step_period)*t)\n", "\n", "def make_kinematic_ref(sinusoid, step_k, scale=0.3, dt=1/50):\n", " \"\"\" \n", " Makes trotting kinematics for the 12 leg joints.\n", " \"\"\"\n", " \n", " _steps = jp.arange(step_k)\n", " step_period = step_k * dt\n", " t = _steps * dt\n", " \n", " wave = sinusoid(t, step_period, scale)\n", " f_leg_cmd_bloc = jp.concatenate(\n", " [jp.zeros((step_k, 1)),\n", " wave.reshape(step_k, 1),\n", " -2*wave.reshape(step_k, 1)],\n", " axis=1\n", " )\n", " h_leg_cmd_bloc = -1 * f_leg_cmd_bloc\n", "\n", " bloc1 = jp.concatenate([\n", " jp.zeros((step_k, 3)),\n", " f_leg_cmd_bloc,\n", " h_leg_cmd_bloc,\n", " jp.zeros((step_k, 3))],\n", " axis=1\n", " )\n", "\n", " bloc2 = jp.concatenate([\n", " f_leg_cmd_bloc,\n", " jp.zeros((step_k, 3)),\n", " jp.zeros((step_k, 3)),\n", " h_leg_cmd_bloc],\n", " axis=1\n", " )\n", "\n", " cycle = jp.concatenate([bloc1, bloc2], axis=0)\n", " return cycle\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "poses = make_kinematic_ref(cos_wave, step_k=25)\n", "\n", "frames = []\n", "init_q = mj_model.keyframe('standing').qpos\n", "mj_data.qpos = init_q\n", "default_ap = init_q[7:]\n", "\n", "for i in range(len(poses)):\n", " mj_data.qpos[7:] = poses[i] + default_ap\n", " mujoco.mj_forward(mj_model, mj_data)\n", " renderer.update_scene(mj_data)\n", " frames.append(renderer.render())\n", "\n", "media.show_video(frames, fps=50, codec='gif')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### The RL Environment" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "def get_config():\n", " def get_default_rewards_config():\n", " default_config = config_dict.ConfigDict(\n", " dict(\n", " scales=config_dict.ConfigDict(\n", " dict(\n", " min_reference_tracking = -2.5 * 3e-3, # to equalize the magnitude\n", " reference_tracking = -1.0,\n", " feet_height = -1\n", " )\n", " )\n", " )\n", " )\n", " return default_config\n", "\n", " default_config = config_dict.ConfigDict(\n", " dict(rewards=get_default_rewards_config(),))\n", "\n", " return default_config\n", "\n", "# Math functions from (https://github.com/jiawei-ren/diffmimic)\n", "def quaternion_to_matrix(quaternions):\n", " r, i, j, k = quaternions[..., 0], quaternions[..., 1], quaternions[..., 2], quaternions[..., 3]\n", " two_s = 2.0 / (quaternions * quaternions).sum(-1)\n", "\n", " o = jp.stack(\n", " (\n", " 1 - two_s * (j * j + k * k),\n", " two_s * (i * j - k * r),\n", " two_s * (i * k + j * r),\n", " two_s * (i * j + k * r),\n", " 1 - two_s * (i * i + k * k),\n", " two_s * (j * k - i * r),\n", " two_s * (i * k - j * r),\n", " two_s * (j * k + i * r),\n", " 1 - two_s * (i * i + j * j),\n", " ),\n", " -1,\n", " )\n", " return o.reshape(quaternions.shape[:-1] + (3, 3))\n", "\n", "def matrix_to_rotation_6d(matrix):\n", " batch_dim = matrix.shape[:-2]\n", " return matrix[..., :2, :].reshape(batch_dim + (6,))\n", "\n", "def quaternion_to_rotation_6d(quaternion):\n", " return matrix_to_rotation_6d(quaternion_to_matrix(quaternion))\n", "\n", "class TrotAnymal(PipelineEnv):\n", "\n", " def __init__(\n", " self,\n", " termination_height: float=0.25,\n", " **kwargs,\n", " ):\n", " step_k = kwargs.pop('step_k', 25)\n", "\n", " physics_steps_per_control_step = 10\n", " kwargs['n_frames'] = kwargs.get(\n", " 'n_frames', physics_steps_per_control_step)\n", "\n", " mj_model = mujoco.MjModel.from_xml_path(xml_path)\n", " mj_model.opt.timestep = 0.002\n", " kp = 230\n", " mj_model.actuator_gainprm[:, 0] = kp\n", " mj_model.actuator_biasprm[:, 1] = -kp\n", "\n", " sys = mjcf.load_model(mj_model)\n", "\n", " super().__init__(sys=sys, **kwargs) \n", " \n", " self.termination_height = termination_height\n", " \n", " self._init_q = mj_model.keyframe('standing').qpos\n", " \n", " self.err_threshold = 0.4 # diffmimic; value from paper.\n", " \n", " self._default_ap_pose = mj_model.keyframe('standing').qpos[7:]\n", " self.reward_config = get_config()\n", "\n", " self.action_loc = self._default_ap_pose\n", " self.action_scale = jp.array([0.2, 0.8, 0.8] * 4)\n", " \n", " self.feet_inds = jp.array([21,28,35,42]) # LF, RF, LH, RH\n", "\n", " #### Imitation reference\n", " kinematic_ref_qpos = make_kinematic_ref(\n", " cos_wave, step_k, scale=0.3, dt=self.dt)\n", " kinematic_ref_qvel = make_kinematic_ref(\n", " dcos_wave, step_k, scale=0.3, dt=self.dt)\n", " \n", " self.l_cycle = jp.array(kinematic_ref_qpos.shape[0])\n", " \n", " # Expand to entire state space.\n", "\n", " kinematic_ref_qpos += self._default_ap_pose\n", " ref_qs = np.tile(self._init_q.reshape(1, 19), (self.l_cycle, 1))\n", " ref_qs[:, 7:] = kinematic_ref_qpos\n", " self.kinematic_ref_qpos = jp.array(ref_qs)\n", " \n", " ref_qvels = np.zeros((self.l_cycle, 18))\n", " ref_qvels[:, 6:] = kinematic_ref_qvel\n", " self.kinematic_ref_qvel = jp.array(ref_qvels)\n", "\n", " # Can decrease jit time and training wall-clock time significantly.\n", " self.pipeline_step = jax.checkpoint(self.pipeline_step, \n", " policy=jax.checkpoint_policies.dots_with_no_batch_dims_saveable)\n", " \n", " def reset(self, rng: jax.Array) -> State:\n", " # Deterministic init\n", "\n", " qpos = jp.array(self._init_q)\n", " qvel = jp.zeros(18)\n", " \n", " data = self.pipeline_init(qpos, qvel)\n", "\n", " # Position onto ground\n", " pen = jp.min(data.contact.dist)\n", " qpos = qpos.at[2].set(qpos[2] - pen)\n", " data = self.pipeline_init(qpos, qvel)\n", "\n", " state_info = {\n", " 'rng': rng,\n", " 'steps': 0.0,\n", " 'reward_tuple': {\n", " 'reference_tracking': 0.0,\n", " 'min_reference_tracking': 0.0,\n", " 'feet_height': 0.0\n", " },\n", " 'last_action': jp.zeros(12), # from MJX tutorial.\n", " 'kinematic_ref': jp.zeros(19),\n", " }\n", "\n", " x, xd = data.x, data.xd\n", " obs = self._get_obs(data.qpos, x, xd, state_info)\n", " reward, done = jp.zeros(2)\n", " metrics = {}\n", " for k in state_info['reward_tuple']:\n", " metrics[k] = state_info['reward_tuple'][k]\n", " state = State(data, obs, reward, done, metrics, state_info)\n", " return jax.lax.stop_gradient(state)\n", " \n", " def step(self, state: State, action: jax.Array) -> State:\n", " action = jp.clip(action, -1, 1) # Raw action\n", "\n", " action = self.action_loc + (action * self.action_scale)\n", "\n", " data = self.pipeline_step(state.pipeline_state, action)\n", " \n", " ref_qpos = self.kinematic_ref_qpos[jp.array(state.info['steps']%self.l_cycle, int)]\n", " ref_qvel = self.kinematic_ref_qvel[jp.array(state.info['steps']%self.l_cycle, int)]\n", " \n", " # Calculate maximal coordinates\n", " ref_data = data.replace(qpos=ref_qpos, qvel=ref_qvel)\n", " ref_data = mjx.forward(self.sys, ref_data)\n", " ref_x, ref_xd = ref_data.x, ref_data.xd\n", "\n", " state.info['kinematic_ref'] = ref_qpos\n", "\n", " # observation data\n", " x, xd = data.x, data.xd\n", " obs = self._get_obs(data.qpos, x, xd, state.info)\n", "\n", " # Terminate if flipped over or fallen down.\n", " done = 0.0\n", " done = jp.where(x.pos[0, 2] < self.termination_height, 1.0, done)\n", " up = jp.array([0.0, 0.0, 1.0])\n", " done = jp.where(jp.dot(math.rotate(up, x.rot[0]), up) < 0, 1.0, done)\n", "\n", " # reward\n", " reward_tuple = {\n", " 'reference_tracking': (\n", " self._reward_reference_tracking(x, xd, ref_x, ref_xd)\n", " * self.reward_config.rewards.scales.reference_tracking\n", " ),\n", " 'min_reference_tracking': (\n", " self._reward_min_reference_tracking(ref_qpos, ref_qvel, state)\n", " * self.reward_config.rewards.scales.min_reference_tracking\n", " ),\n", " 'feet_height': (\n", " self._reward_feet_height(data.geom_xpos[self.feet_inds][:, 2]\n", " ,ref_data.geom_xpos[self.feet_inds][:, 2])\n", " * self.reward_config.rewards.scales.feet_height\n", " )\n", " }\n", " \n", " reward = sum(reward_tuple.values())\n", "\n", " # state management\n", " state.info['reward_tuple'] = reward_tuple\n", " state.info['last_action'] = action # used for observation. \n", "\n", " for k in state.info['reward_tuple'].keys():\n", " state.metrics[k] = state.info['reward_tuple'][k]\n", "\n", " state = state.replace(\n", " pipeline_state=data, obs=obs, reward=reward,\n", " done=done)\n", " \n", " #### Reset state to reference if it gets too far\n", " error = (((x.pos - ref_x.pos) ** 2).sum(-1)**0.5).mean()\n", " to_reference = jp.where(error > self.err_threshold, 1.0, 0.0)\n", "\n", " to_reference = jp.array(to_reference, dtype=int) # keeps output types same as input. \n", " ref_data = self.mjx_to_brax(ref_data)\n", "\n", " data = jax.tree_util.tree_map(lambda x, y: \n", " jp.array((1-to_reference)*x + to_reference*y, x.dtype), data, ref_data)\n", " \n", " x, xd = data.x, data.xd # Data may have changed.\n", " obs = self._get_obs(data.qpos, x, xd, state.info)\n", " \n", " return state.replace(pipeline_state=data, obs=obs)\n", " \n", " def _get_obs(self, qpos: jax.Array, x: Transform, xd: Motion,\n", " state_info: Dict[str, Any]) -> jax.Array:\n", "\n", " inv_base_orientation = math.quat_inv(x.rot[0])\n", " local_rpyrate = math.rotate(xd.ang[0], inv_base_orientation)\n", "\n", " obs_list = []\n", " # yaw rate\n", " obs_list.append(jp.array([local_rpyrate[2]]) * 0.25)\n", " # projected gravity\n", " obs_list.append(\n", " math.rotate(jp.array([0.0, 0.0, -1.0]), inv_base_orientation))\n", " # motor angles\n", " angles = qpos[7:19]\n", " obs_list.append(angles - self._default_ap_pose)\n", " # last action\n", " obs_list.append(state_info['last_action'])\n", " # kinematic reference\n", " kin_ref = self.kinematic_ref_qpos[jp.array(state_info['steps']%self.l_cycle, int)]\n", " obs_list.append(kin_ref) # Gait schedule, per actuator.\n", "\n", " obs = jp.clip(jp.concatenate(obs_list), -100.0, 100.0)\n", "\n", " return obs\n", " \n", " def mjx_to_brax(self, data):\n", " \"\"\" \n", " Apply the brax wrapper on the core MJX data structure.\n", " \"\"\"\n", " q, qd = data.qpos, data.qvel\n", " x = Transform(pos=data.xpos[1:], rot=data.xquat[1:])\n", " cvel = Motion(vel=data.cvel[1:, 3:], ang=data.cvel[1:, :3])\n", " offset = data.xpos[1:, :] - data.subtree_com[self.sys.body_rootid[1:]]\n", " offset = Transform.create(pos=offset)\n", " xd = offset.vmap().do(cvel)\n", " data = _reformat_contact(self.sys, data)\n", " return data.replace(q=q, qd=qd, x=x, xd=xd)\n", "\n", "\n", " # ------------ reward functions----------------\n", " def _reward_reference_tracking(self, x, xd, ref_x, ref_xd):\n", " \"\"\"\n", " Rewards based on inertial-frame body positions.\n", " Notably, we use a high-dimension representation of orientation.\n", " \"\"\"\n", "\n", " f = lambda x, y: ((x - y) ** 2).sum(-1).mean()\n", "\n", " _mse_pos = f(x.pos, ref_x.pos)\n", " _mse_rot = f(quaternion_to_rotation_6d(x.rot),\n", " quaternion_to_rotation_6d(ref_x.rot))\n", " _mse_vel = f(xd.vel, ref_xd.vel)\n", " _mse_ang = f(xd.ang, ref_xd.ang)\n", "\n", " # Tuned to be about the same size.\n", " return _mse_pos \\\n", " + 0.1 * _mse_rot \\\n", " + 0.01 * _mse_vel \\\n", " + 0.001 * _mse_ang\n", "\n", " def _reward_min_reference_tracking(self, ref_qpos, ref_qvel, state):\n", " \"\"\" \n", " Using minimal coordinates. Improves accuracy of joint angle tracking.\n", " \"\"\"\n", " pos = jp.concatenate([\n", " state.pipeline_state.qpos[0:3],\n", " state.pipeline_state.qpos[4:]])\n", " pos_targ = jp.concatenate([\n", " ref_qpos[0:3],\n", " ref_qpos[4:]])\n", " pos_err = jp.linalg.norm(pos_targ - pos)\n", " vel_err = jp.linalg.norm(state.pipeline_state.qvel- ref_qvel)\n", "\n", " return pos_err + vel_err\n", "\n", " def _reward_feet_height(self, feet_pos, feet_pos_ref):\n", " return jp.sum(jp.abs(feet_pos - feet_pos_ref)) # try to drive it to 0 using the l1 norm.\n", "\n", "envs.register_environment('trotting_anymal', TrotAnymal)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Imitation Learning via FoPG's\n", "Takes 15 minutes on a NVIDIA 3060 TI GPU" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "make_networks_factory = functools.partial(\n", " apg_networks.make_apg_networks,\n", " hidden_layer_sizes=(256, 128)\n", ")\n", "\n", "epochs = 500\n", "\n", "train_fn = functools.partial(apg.train,\n", " episode_length=240,\n", " policy_updates=epochs,\n", " horizon_length=32,\n", " num_envs=64,\n", " learning_rate=1e-4,\n", " num_eval_envs=64,\n", " num_evals=10 + 1,\n", " use_float64=True,\n", " normalize_observations=True,\n", " network_factory=make_networks_factory)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAisAAAGdCAYAAADT1TPdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABBvElEQVR4nO3deXwU9eH/8ffm2twXCzkgJ3e4LyGAIqKAUqwXlmoptohfK6kooIJQuURUwH6/xSq1trS/ql9PPJBa4QvUitynQAQEEghHAkkgCwnZHDu/P4CtKWcgm9ndvJ6PxzzYnZmdvLOQ7JuZz8xYDMMwBAAA4KH8zA4AAABwOZQVAADg0SgrAADAo1FWAACAR6OsAAAAj0ZZAQAAHo2yAgAAPBplBQAAeLQAswNcL6fTqSNHjigiIkIWi8XsOAAA4CoYhqFTp04pMTFRfn6X33fi9WXlyJEjSkpKMjsGAAC4Bnl5eWrWrNll1/H6shIRESHp7DcbGRlpchoAAHA17Ha7kpKSXJ/jl+P1ZeX8oZ/IyEjKCgAAXuZqhnAwwBYAAHg0ygoAAPBolBUAAODR3FZWcnNzNWrUKKWlpSkkJETNmzfX1KlTVVFRUWM9i8VywbRgwQJ3xQIAAF7GbQNsd+3aJafTqT/84Q9q0aKFduzYodGjR6u0tFRz586tse7ChQs1ePBg1/OoqCh3xQIAAF7GbWVl8ODBNQpIenq6du/erddff/2CshIdHa34+Hh3RQEAAF6sXseslJSUKDY29oL5WVlZstls6tGjhxYsWCCn03nJbTgcDtnt9hoTAADwXfV2nZV9+/Zp/vz5mjdvXo35M2fO1IABAxQSEqLly5dr/PjxKiws1JQpUy66ndmzZ2v69On1ERkAAHgAi2EYRm1eMG3atCuWhQ0bNqh79+6u50eOHFG/fv3Ur18/vfnmm5d97bx58zRjxgyVlJRcdLnD4ZDD4XA9P38FvJKSEi4KBwCAl7Db7YqKirqqz+9al5XCwkIVFhZedp3U1FQFBwdLOltU+vfvr549e+ovf/nLFW9W9M0336hv377Kz89XXFzcFfPU5psFAACeoTaf37U+DGSz2WSz2a5q3cOHD6t///7q1q2bFi5ceMWiIklbtmxRcHCwoqOjaxsNAAD4ILeNWTly5IhuvvlmJScna+7cuTp+/Lhr2fkzfxYvXqz8/HxlZmYqJCREK1eu1OTJk/XII4/IarW6KxoAAPAibisrS5cu1d69e7V3794Lbv18/shTYGCgXnvtNY0bN05Op1Pp6emaMWOGxowZ465YAADAy9R6zIqnYcwKAADuUVZRpYznvpQkZc8YpNCgutvHUZvPb+4NBAAAPBplBQAAeDTKCgAA8GiUFQAA4NEoKwAAwKNRVgAAgEejrAAAAI9GWQEAAB6NsgIAADwaZQUAAHg0ygoAwKuUVVQpdeISpU5corKKKrPjoB5QVgAAgEejrAAAAI9GWQEAAB6t7u71DAAAfILTaeirPcf15tf7zY4iibICAADOOe2o0ocb8/TXNQeUU1hqdhwXygoAAG5WVlGljOe+lCRlzxik0CDP+vg9UFSqv6zO1QcbD+m04+wZVhHBAbq3azP9ZXWuueFEWQEAoEEyDEPf7C3SX1bnaPmuYzKMs/PTG4fpF71TdU/XZrJYRFkBAAD160xFtT7eclh/WZ2jPQWnXfNvbt1Yv+iTphtb2OTnZ5Ekj7mODWUFAIAG4NCJMv1t7QG9uz5PJWcqJUlhQf66r1szjeydqvTG4SYnvDTKCgAAPsowDG3IPaGF3+Toy535cp471JMcG6qRvVM1rHszRQYHmhvyKlBWAADwMeWV1Vq87Yj+sjpXO4/YXfP7tGikh3qn6ZY2TeR/7lCPN6CsAADgIwrs5Xpr7QG9s+6gikorJEnWAD/d07WpHuqdptbxESYnvDaUFQAAvNyWgye08Jtc/X37UVWdO9aTGBWsEZmpGt4jSTFhQSYnvD6UFQAAvFBFlVNf7Diqhd/kamveSdf8Hqkx+kWfNA3MiFOAv2/cVYeyAgCAFyk87dA76w7qrbUHdOyUQ5IU5O+noZ0S9Ys+qWrfNMrkhHWPsgIAgBfYeaREC7/J1WfbjqiiyilJahxh1YheKfrpDclqHGE1OaH7UFYAAPBQVdVOLcsu0MJvcrU+t9g1v1OzKP2iT5ru6JCgoADfONRzOZQVAAA8zMmyCr27IU9/W3NAh0+ekSQF+Fl0e4cE/aJPqromx5icsH5RVgAA8BB7Ck5p4Te5+njLIZVXnj3UExsWpAduSNbPeqUoPirY5ITmoKwAAGAip9PQil3HtHB1jr7ZW+Sa3zYhUr/ok6o7OyUqONDflGyhQQHKfXGIKV/7hygrAACY4FR5pd7feEj/b02uDhSVSZL8LNLAjHg91CdVPdNiZbF4z1Vm3YmyAgBAPcotLNX7Gw/pg415Kq2oliRFBgdo+A3JGtErRUmxoSYn9DyUFQAA3MgwDOUUlrqe3/G7Va7HLZqE66Heqbqna1OFBvGRfClufWfuvPNObd26VceOHVNMTIxuvfVWvfTSS0pMTHStc/DgQY0ZM0YrVqxQSEiIHnjgAc2dO1dBQd59aWAAQMNU7TS0K9+u9TnFWp9TrA25xSo8XVFjnVvaNNEv+qSqbwsbh3quglvLSv/+/fXss88qISFBhw8f1oQJE3Tfffdp9erVkqTq6moNGTJEjRs31qpVq1RUVKSRI0fKMAzNnz/fndEAAKgTFVVObT9ccq6cFGlj7gmdclTVWCcowM91Ibe/j+2rjATfu8qsO7m1rDz55JOuxykpKZo4caLuuusuVVZWKjAwUEuXLlV2drby8vJce1vmzZunhx56SLNmzVJkZKQ74wEAUGtnKqq15eAJrTu352RL3gnXacbnhVsD1C0lRjekxapnWqxaNAlX5xnLJEmpjcLMiO3V6u0AWXFxsd5++2317t1bgYGBkqQ1a9aoffv2NQ4LDRo0SA6HQ5s2bVL//v0v2I7D4ZDD4XA9t9vt7g8PAGiwSsoqtfFAsdbnni0n2w+VuO5sfF5sWJB6pMbohrRG6pkWqzbxETVuIlhWUfWfm0UtuL2sPPPMM3r11VdVVlamXr166fPPP3cty8/PV1xcXI31Y2JiFBQUpPz8/Itub/bs2Zo+fbpbMwPAtSirqFLGc19KkrJnDPKKAZPemNndjp0q14acE9qQW6x1OcXalW+XUbObKCEqWDekxbr2nDRvHM7YEzeq9b/KadOmXbEsbNiwQd27d5ckPfXUUxo1apQOHDig6dOn6+c//7k+//xz11/qxf5yDcO45F/6pEmTNG7cONdzu92upKSk2n4bAADIMAwdOnHGNRB2fU6x9v/gzJ3z0m1h6pEa6yoozWJCKCf1qNZlJSsrS8OHD7/sOqmpqa7HNptNNptNrVq1Utu2bZWUlKS1a9cqMzNT8fHxWrduXY3XnjhxQpWVlRfscTnParXKavXdO0sCANzHMAztO37aNd5kQ06xjpSU11jHYpHaxEfqhnOHdXqkxahJRMO8zL2nqHVZOV8+roVxbj/a+TEnmZmZmjVrlo4ePaqEhARJ0tKlS2W1WtWtW7dr+hoAAJxX7TT03VH7uXJSpA25J1RcWvM04gA/izo0i9IN5/acdE+JVVRooEmJcTFuOzi5fv16rV+/Xn379lVMTIz279+v5557Ts2bN1dmZqYkaeDAgcrIyNCIESM0Z84cFRcXa8KECRo9ejRnAgEAas1RVa3th0pce042H7jwNGJrgJ+6Jse4Dul0SY5mrI6Hc9vfTkhIiBYtWqSpU6eqtLRUCQkJGjx4sN59913XYRx/f38tWbJEjz32mPr06VPjonAAAFxJqaNKmw+c1PqcIq3LKdbWvJNyVNU8jTjCGqDu5w7p3JAWow5NoxUU4HeJLcITua2sdOjQQStWrLjiesnJyTXOEAIA4HJKf7CnpNfsFar+j9OIG4UFufaa9EiNVduESPn7MRjWm7HfCwDgFQ6fPKO/rs7VO+sOuuZVOw01jQ6pUU6aNw7jTB0fQ1kBAHi0bXkn9eaqHP19+9EL9qIsG3eTWjaJMCkZ6gtlBQDgcaqdhpZlF+hPq/ZrQ+4J1/zezRtpRK8U/ertzZKkptEhZkVEPaKsAAA8RqmjSh9szNOfv8nVweIySVKgv0VDOyVqVN80tUuM4tL1DRBlBQBguqMlZ/SXc+NRTpWfLSPRoYF6sGeyfp6ZqrhILsrWkFFWAHgk7lnTMHx76KT+tCpHS7496ro5YJotTL/sm6Z7uzbl7x2SKCsAgHpW7TS0/LsCvbkqR+tzil3ze6XH6uG+6bqlTRP5caoxfoCyAgCoF2UVVfpw0yH9eVWOcovOjkcJ8LPozk6J+mXfNLVvGmVyQngqygoAwK3yS8r11zVnx6OUnKmUJEWF/Hs8SnyU749HCQ0KUO6LQ8yO4bUoKwAAt9hxuER/WpWjxduOuMajpDYK1ai+abq3WzPGo+Cq8S8FAFBnnE5Dy3cd05tf79e6H4xH6ZkWq4dvPDsehUvfo7YoKwBQRyp+cAO9jbkn1KlZtKJCA01MVH/KKqr00aZD+vM3ucopLJV0djzKjzomaFTfdHVoxngUXDvKCgBcB3t5pVbuOqal2QVaueuYa/7P/7xekhQXaVXr+Ei1jgtXq7gItY6PUMsmEQoJ8jcrcp0qsJfrr6tz9fYPxqNEBgfogZ4pGtk7RQlRXGEW14+yAgC1dMxermXfFejLnQVas69QldXGBeskRgfryMlyFdgdKrAf17/2HHcts1iklNhQtYqLUJv4CLWKj1DruAil2sIU6O9Xn9/KNdt5pER/+jpHi7894vr+UxqF6pd90nRft2YKs/LxgrrDvyYAuAr7j5/W0uwCfbkzX1sOnqyxrHnjMA1qF6+bWtk0/I11kqT/G9dP1U5D3x87rd35p7Q7/5T2FJz9s6i0QrlFZcotKtPS7ALXdoL8/ZTeOEyt48/ugWkdF6FWcRFqGh3iEdcdcToNrdx9TG9+naM1+4tc829IjdWoG9N0a9s4xqPALSgrAHARhmHo20MlWpqdry93FmjvsdM1lndOitagdvG6LSNOLZqES9IF96yJCA5U1+QYdU2OqTG/8LRDe/JPadf5AlNwSnvyT6m0olq7zs3/obAgf9felx/ujbGFW93wnV/oTEW1Ptp89voo+8+NR/H3s2hIhwSN6pumTknR9ZIDDRdlBQDOqax2at3+Yi3NztfSnQXKt5e7lgX4WZTZvJGroFzPvWps4VbZWljVu4XNNc/pNHT45BlXeTm/N2bf8dMqrajWloMnL9ij0ygsSK3jI1xjYc4/Dq+jQzDH7OX6f2sO6K11B3Sy7Ox4lIjgAD1wQ7JG9k5VInc8Rj2hrABo0MoqqvTV7uNaml2g5d8VyF7+770jYUH+url1Ew1sF6ebWzdRVIj7zuzx87MoKTZUSbGhGtA2zjW/stqp3MJS196X83tjDhSXqai0Qqv3FWn1vqIa22oaHVJjLEyruAg1bxIma8DVDerNPmLXn1bl6LNth13jUZJiQ/TLPmka1j2pzsoQcLX4FwegwSk67dDyXce0dGe+vv6+UI4fnHLcKCxIt2XEaWC7OPVublNwoLln7QT6+6llXIRaxkVIHf89/0xFtb4/9u+xMOdLTIHdocMnz+jwyTNa/oOzk/z9LEqzhal1XESNvTG28CDXOl/tOa631h7QN3v/XX66p8To4RvTdFtGPONRYBrKCoAGIa+4zDVAdmNusZw/OIEnOTZUg9rFaWC7eHVNjvGKD+WQIH91bBatjs2ia8w/WVahPQWntTvffm5vzGntyrfLXl6lvcdOa++x01qy/ahr/eDAf5999Ku3Nks6W2zuODcepTPjUeABKCsAfJJhGPru6CnXANnvjtprLG/fNFIDM+I1sF2cWsdFyGLx/IJyNaJDg3RDWqxuSIt1zTMMQwV2x7mxMHbtzj+tPQVn98SUV/57r1K4NUAP9Dw7HqUp41HgQSgrAHxGtdPQxtxiLc0u0NLsfOUVn3Et87NIN6TFugbINosJNTFp/bJYLIqPClZ8VLD6tWrsml/tNLS7wK47/meVJGnF+H5qch0DhwF3oawA8GrlldVa9X2hlmbn6/++O6bi0grXMmuAn25q1ViD2sXrljZNFBsWdJktNTz+fhalNgpzPQ8P5iMBnol/mQC8TklZpVbuPqYvd+brqz3HVVZR7VoWFRKoAW2baFC7eN3Y0sadfQEfwE8xAK+QX1KuZefGn6zdX6SqH4yQTYwK1sB28RqYEaceabFec8l6AFeHsgLA4/3kD2u0/XDNAbKt4yI0sF2cBmbEq33TSJ8ZIAvgQpQVAB6nosqpBf/c53q+/bBdFovUNTlGg9rF6baMeKXZwi6zBQC+hLICwKNsyC3Ws4u26/sf3Itn+p0Zur1DgppEcKYK0BBRVgB4hJKySr34j1363/UHJUmxYUGuM3uGdU9ioCzQgDEKDYCpDMPQ4m1HNOCVr1xF5Sfdk/T5r/uYnAyAp+C/KgBMk1dcpt98ukP/3H1ckpTeOEyz7+6gnumNVFZRdYVXA2goKCsA6l1VtVMLv8nVK8v26ExltYL8/fRY/+b61c3Nr/rOwAAaDsoKgHr17aGTmrRou3YeOXsqcs+0WM26u4NaNAk3ORkAT+XWMSt33nmnkpOTFRwcrISEBI0YMUJHjhypsY7FYrlgWrBggTtjAQ1OWUWVUicuUerEJaYdXjntqNL0xTt11++/0c4jdkWFBOrlezvq3Ud6+UxRCQ0KUO6LQ5T74hAGBAN1yK0/Tf3799ezzz6rhIQEHT58WBMmTNB9992n1atX11hv4cKFGjx4sOt5VFSUO2MBqGfLsgv03Kc7dLSkXJL0486J+s2PMmQLt5qcDIA3cGtZefLJJ12PU1JSNHHiRN11112qrKxUYGCga1l0dLTi4+PdGQWACfJLyjXts536x858SVJSbIiev6tDjTv/AsCV1Nt+yuLiYr399tvq3bt3jaIiSVlZWXr44YeVlpamUaNG6ZFHHpGf38WPUDkcDjkcDtdzu91+0fUAmKfaaejtdQf08j9267SjSv5+Fo2+MV1jB7RUSBADaHF9zh9uQ8Ph9rLyzDPP6NVXX1VZWZl69eqlzz//vMbymTNnasCAAQoJCdHy5cs1fvx4FRYWasqUKRfd3uzZszV9+nR3xwZwjb47atekRdu1Ne+kJKlzUrRm39NBbRMizQ0GwGvVeoDttGnTLjoo9ofTxo0bXes/9dRT2rJli5YuXSp/f3/9/Oc/l2H8+26pU6ZMUWZmpjp37qzx48drxowZmjNnziW//qRJk1RSUuKa8vLyavstAHCDMxXVevGLXRo6f5W25p1UuDVAM37cTh/9qjdFBcB1qfWelaysLA0fPvyy66Smproe22w22Ww2tWrVSm3btlVSUpLWrl2rzMzMi762V69estvtKigoUFxc3AXLrVarrFYG5QGe5F97jmvKJzt0sLhMkjS4Xbym3dlO8VHcywfA9at1WTlfPq7F+T0qPxxz8p+2bNmi4OBgRUdHX9PXAFB/Ck87NPPzbH269ewlCRKigjXjx+11W8aF/9EAgGvltjEr69ev1/r169W3b1/FxMRo//79eu6559S8eXPXXpXFixcrPz9fmZmZCgkJ0cqVKzV58mQ98sgj7D0BPJhhGPpg4yHN+vt3KjlTKT+LNLJ3qsYPbK1wK9cXAVC33PZbJSQkRIsWLdLUqVNVWlqqhIQEDR48WO+++66riAQGBuq1117TuHHj5HQ6lZ6erhkzZmjMmDHuigXgOu07flrPLtqudTnFkqSMhEi9eG8HdWwWbW4wAD7LbWWlQ4cOWrFixWXXGTx4cI2LwQHwXI6qar3+z316beU+VVQ7FRLor3G3tdIv+qQqwJ8buANwH/bXAriidfuL9OzH27XveKkkqX/rxprx4/ZKig01ORmAhoCyAuCSTpZVaPbfd+m9jWcvEWALt2ranRka0iFBFovF5HQAGgrKCoALGIahz7Yd0czPs1V4ukKS9EDPZD0zuI2iQgKv8GoAqFuUFQA1HCwq05RPd+hfe45Lklo2CdcL93RQj9RYk5PBHbh0PbwBZQWAJKmy2qk3v87R/yzfo/JKp4IC/PTr/i30X/2aKyig/gfQ8iEK4DzKCgBtOXhCkxZt1678U5KkzPRGmnV3e6U3Djc5GQBQVoAG7VR5peZ8uVt/W3tAhiHFhAZq8pAM3du1KQNoAXgMygrQQP1jR76mfbZT+fZySdI9XZtqypAMxYYFmZwMAGqirAANTH5JuWZ/sUvLsgskSamNQjXr7g7q0+La7vkFAO5GWQEamB/NX6WyimoF+Fn0aL/myrqlhYID/c2OBQCXRFkBGoCKKqfrcVlFtbqlxGj2PR3UKi7CxFQAcHUoK0AD8Po/97keTx2aoZGZqfLzYwAtAO/A3ccAH7ct76TeXJXjev6THkkUFQBehbIC+LDyymqN/2Cbqp2G2VEA4JpRVgAfNm/pbu09dlq2cE5HBuC9KCuAj9qQW+w6/DPjx+1MTgMA146yAvigsooqTfhgmwxDGtatmW5u3cTsSABwzSgrgA966YtdOlBUpsSoYP1maIbZcQDgulBWAB/zzd5C/XXNAUnSS/d1VGRwoMmJAOD6UFYAH3KqvFJPf/itJOlnvZJ1Y8vGJicCgOtHWQF8yKwl3+nwyTNKjg3VpNvbmh0HAOoEZQXwESt3HdO7G/JksUhz7uuoMCsXqAbgGygrgA8oKavUxEVnD//8oneaeqY3MjkRANQdygrgA6Yt3qkCu0PpjcP09ODWZscBgDpFWQG83D925OvjLYflZ5HmDeuk4EB/syMBQJ2irABerOi0Q5M/3i5J+q9+zdUlOcbkRABQ9ygrgJcyDEO/+XSHikor1DouQk/c2tLsSADgFpQVwEst/vao/r49XwF+Fs27v5OsARz+AeCbKCuAFzpmL9dzn+6QJGXd0kLtm0aZnAgA3IeyAtRSWUWVUicuUerEJSqrqKr3r28YhiYt2q6TZZVq3zRSY/q3qPcMAFCfKCuAl/lw0yEt33VMQf5+mjesswL9+TEG4Nv4LQd4kSMnz2jG4mxJ0pO3tVLr+AiTEwGA+1FWAC9hGIae+ehbnXJUqUtytB65Kd3sSABQLygrgJd4e91Bff19oawBfpo7rJP8/SxmRwKAelEvZcXhcKhz586yWCzaunVrjWUHDx7U0KFDFRYWJpvNpscff1wVFRX1EQvwGgeLyvTC37+TJD0zuI2aNw6v1etDgwKU++IQ5b44RKFB3OAQgHepl99aTz/9tBITE7Vt27Ya86urqzVkyBA1btxYq1atUlFRkUaOHCnDMDR//vz6iAZ4PKfT0IQPt6msolo902L1UO9UsyMBQL1y+56VL774QkuXLtXcuXMvWLZ06VJlZ2frrbfeUpcuXXTrrbdq3rx5+uMf/yi73e7uaIBXWLg6V+tzihUa5K8593WSH4d/ADQwbi0rBQUFGj16tP72t78pNDT0guVr1qxR+/btlZiY6Jo3aNAgORwObdq06aLbdDgcstvtNSbAV+07flov/2OXJOnZO9oqudGFP0cA4OvcVlYMw9BDDz2kRx99VN27d7/oOvn5+YqLi6sxLyYmRkFBQcrPz7/oa2bPnq2oqCjXlJSUVOfZAU9QVe3UhA+2yVHl1I0tbXqwZ7LZkQDAFLUuK9OmTZPFYrnstHHjRs2fP192u12TJk267PYslgt3aRuGcdH5kjRp0iSVlJS4pry8vNp+C4BXeOPr/dpy8KQirAF66d6Ol/yZAABfV+sBtllZWRo+fPhl10lNTdXzzz+vtWvXymq11ljWvXt3Pfjgg/rrX/+q+Ph4rVu3rsbyEydOqLKy8oI9LudZrdYLtgn4mt35p/Tfy76XJD03NEOJ0SEmJwIA89S6rNhsNtlstiuu97vf/U7PP/+86/mRI0c0aNAgvffee+rZs6ckKTMzU7NmzdLRo0eVkJAg6eygW6vVqm7dutU2GuATKqudGvf+VlVUOzWgTRPd162Z2ZEAwFRuO3U5Obnm8fXw8LPXhWjevLmaNTv7y3fgwIHKyMjQiBEjNGfOHBUXF2vChAkaPXq0IiMj3RUN8Gi/X7lXO4/YFR0aqNn3dODwD4AGz9Qr2Pr7+2vJkiUKDg5Wnz59dP/99+uuu+666GnOQEOw43CJXl2xV5I048ft1SQy2OREAGC+eruUZWpqqgzDuGB+cnKyPv/88/qKAXgsR1W1xr2/VVVOQ3d0iNfQjglmRwIAj8C9gQAP8d//9732FJyWLTxIM3/cnsM/AHAOZQXwAJsPntAfvtonSZp1dwc1CueMNwA4j7ICmOxMRbUmvL9NTkO6u0tTDWoXb3YkAPAolBXAZHO+3K39haWKi7Rq2tB2ZscBAI9DWQFMtHZ/kRauzpEkvXhvR0WFBpqcCAA8D2UFMEmpo0pPfbhNhiEN75Gk/q2bmB0JADwSZQUwyQt//055xWfUNDpEk4e0NTsOAHgsygpggq+/P6631x2UJM25r6Migjn8AwCXQlkB6pm9vFJPf/itJGlkZop6t7jyvbYAoCGjrAD1bMbibB0tKVdqo1A9c3sbs+MAgMejrAD1aPl3Bfpw0yFZLNLcYZ0UGlRvd7wAAK9FWQHqyYnSCk1ctF2S9HDfNHVPjTU5EQB4B8oKUE+e+2ynjp9yqEWTcI0f2NrsOADgNSgrQD34+/ajWrztiPz9LJo3rJOCA/3NjgQAXoOyArhZ4WmHpnyyQ5L0q37N1Skp2txAAOBlKCuAGxmGoWcXbVdxaYXaxEfo8QEtzY4EAF6HsgJTlVVUKXXiEqVOXKKyiiqz49S5T7Ye1tLsAgX6W/TK/Z0VFMCPHADUFr85ATfJLynX1E93SpIev6WlMhIjTU4EAN6JsgK4gWEYmrjoW9nLq9SxWZR+dXNzsyMBgNeirABu8N6GPP1z93EFBfhp3rBOCvDnRw0ArhW/QYE6duhEmZ5f8p0kacLAVmoZF2FyIgDwbpQVoA45nYae/vBbnXZUqXtKjEb1TTc7EgB4PcoKUIfeWndAq/cVKTjQT3OGdZK/n8XsSADg9SgrQB3JLSzV7L/vkiRNur2t0mxhJicCAN9AWQHqQLXT0IQPtulMZbUy0xtpRK8UsyMBgM+grAB14M+rcrTxwAmFBfnr5fs6yo/DPwBQZygrwHXae+yU5izdLUma8qMMJcWGmpwIAHwLZQW4DlXVTo1/f5sqqpzq16qxhvdIMjsSAPgcygpwHd5claNth0oUGRygl+7tKIuFwz8AUNcoK8B1eO2f+yRJ0+5sp/ioYJPTAIBvoqwA16Gq2tBtGXG6u0tTs6MAgM+irAC1UFZRpf3HT7ueR4cG6oW7O3D4BwDcKMDsAICncFRVq6DEoSMlZ3S05IyOnCx3/Xnk5BkdLSlXyZnKGq957kcZahxhNSkxADQMlBU0CNVOQ8dOlbsKyNGT5TpScsZVQo6cLFfhacdVbSvM6q9SR7UkaXD7eHfGBgConsqKw+FQz549tW3bNm3ZskWdO3d2LbvY7vPXX39djz76aH1Egw8wDENFpRWuAnL0XAE5fO7PoyfPqOCUQ9VO44rbsgb4KTE6RAlRwUqIClFi9Nk/E6KDlXjuzwA/izKe+7IevjMAgFRPZeXpp59WYmKitm3bdtHlCxcu1ODBg13Po6Ki6iMWvIBhGLKXV9XYG3L03GGZs4drynW0pFwVVc4rbsvfz6L4yOALC0hUsKugxIYFXXH8SVlFVV19ewCAq+D2svLFF19o6dKl+uijj/TFF19cdJ3o6GjFx7M7vaFbtPmwik5X6GjJmRp7RUorqq/4WotFahxuVUJ0iBIvsVekcYSVuyADgBdya1kpKCjQ6NGj9cknnyg09NKXIM/KytLDDz+stLQ0jRo1So888oj8/C5+opLD4ZDD8e+xBXa7vc5zo/58e+ik6/GUT3Zccr2Y0MCLH5Y5t1ckLjJYQQGc3AYAvshtZcUwDD300EN69NFH1b17d+Xm5l50vZkzZ2rAgAEKCQnR8uXLNX78eBUWFmrKlCkXXX/27NmaPn26u2KjHp2pqNakRf8uKH1bNFKzmNALxogkRoUoJMjfxKQAADPVuqxMmzbtimVhw4YNWr16tex2uyZNmnTZdX9YSs4PvJ0xY8Yly8qkSZM0btw413O73a6kJO7H4o1e+scu5RSWup6/8fPuCg3iBDUAQE21/mTIysrS8OHDL7tOamqqnn/+ea1du1ZWa81rUHTv3l0PPvig/vrXv170tb169ZLdbldBQYHi4uIuWG61Wi/YJrzPN3sL9ZfVuWbHAAB4gVqXFZvNJpvNdsX1fve73+n55593PT9y5IgGDRqk9957Tz179rzk67Zs2aLg4GBFR0fXNhq8RMmZSk344OyZYcN7JOndDXkmJwIAeDK37XNPTk6u8Tw8PFyS1Lx5czVr1kyStHjxYuXn5yszM1MhISFauXKlJk+erEceeYS9Jz5s+uKdOlpSrpRGoZowqBVlBQBwWaYOEAgMDNRrr72mcePGyel0Kj09XTNmzNCYMWPMjAU3+seOfC3afFh+FmnesE6MUQEAXFG9fVKkpqbKMGpeQXTw4ME1LgYH31Z42qHJH2+XJP1Xv+bqnhrLBdYAAFfEhSlQLwzD0KRF21VUWqE28RF64taWZkcCAHgJygrqxYebDmlZdoEC/S367U86yxrAdVMAAFeHsgK3O3SiTNMXZ0uSxt3WWm0TIk1OBADwJpQVuJXTaWjCB9t02lGlbikxeuSmdLMjAQC8DGUFbrVwda7W7i9WSKC/5g3rxI0EAQC1RlmB2+w9dkov/WOXJGnykLZKtYWZnAgA4I0oK3CLymqnxr2/TRVVTt3UqrEe7Jl85RcBAHARlBW4xe9X7tW3h0oUFRKol+/tKIuFwz8AgGtDWUGd+/bQSc1fsVeSNPOu9oqPCjY5EQDAm1FWUKfKK6v15HtbVe009KOOCbqzU6LZkQAAXo6ygjr18j92a9/xUjWJsGrmj9ubHQcA4AMoK6gzq/cV6s/f5EiSXrq3o2LCgkxOBADwBZQV1Al7eaWe+uBbSdJPb0hW/zZNTE4EAPAVlBXUiZmLs3X45Bklx4ZqypC2ZscBAPgQygqu29Kd+fpg0yFZLNK8+zspzBpgdiQAgA/hUwXXpfC0Q5MWbZckPXJTunqkxtbq9aFBAcp9cYg7ogEAfAR7VnDNDMPQ5I+3q6i0Qq3jIjTutlZmRwIA+CDKCq7Zos2H9eXOAgX6W/TKTzrJGuBvdiQAgA+irOCaHD55RtM+2ylJeuLWVmqXGGVyIgCAr6KsoNacTkNPfbBNpxxV6pIcrf+6Kd3sSAAAH0ZZQa39vzW5Wr2vSCGB/nrl/s4K8OefEQDAffiUQa3sPXZas7/YJUl69o42SrOFmZwIAODrKCu4alXVTo1/f6scVU7d2NKmn/VKMTsSAKABoKzgqr32z33adqhEkcEBevm+jrJYLGZHAgA0AJQVXJXth0r0u+XfS5Jm/Li9EqJCTE4EAGgoKCu4ovLKaj35/lZVOQ3d0SFeP+6caHYkAEADQlnBFc39crf2HjstW7hVz9/VgcM/AIB6RVnBZa3dX6Q/fZMjSXrp3g6KDQsyOREAoKGhrOCSTpVXasIH22QY0vAeSRrQNs7sSACABoiygkt6/vPvdOjEGTWLCdGUH2WYHQcA0EBRVnxIWUWVUicuUerEJSqrqLqubf1fdoHe25gni0WaN6yTwq0BdZQSAIDaoazgAkWnHZq46FtJ0sN909QzvZHJiQAADRllBTUYhqHJH+9Q4ekKtWwSrvEDW5sdCQDQwFFWUMMnWw/rHzvzFeBn0W9/0lnBgf5mRwIANHBuLSupqamyWCw1pokTJ9ZY5+DBgxo6dKjCwsJks9n0+OOPq6Kiwp2xcAlHS87ouU93SpLGDmip9k2jTE4EAIDk9lGTM2bM0OjRo13Pw8PDXY+rq6s1ZMgQNW7cWKtWrVJRUZFGjhwpwzA0f/58d0fDDzidhp764FudKq9Sp6Ro/erm5mZHAgBAUj2UlYiICMXHx1902dKlS5Wdna28vDwlJp69hPu8efP00EMPadasWYqMjHR3PJzz1roDWrW3UMGBfnrl/k4K8OcIIQDAM7j9E+mll15So0aN1LlzZ82aNavGIZ41a9aoffv2rqIiSYMGDZLD4dCmTZsuuj2HwyG73V5jwvXZf/y0Xvj7d5KkSbe3VfPG4Vd4RcMWGhSg3BeHKPfFIQoN4pRuAHA3t/6mHTt2rLp27aqYmBitX79ekyZNUk5Ojt58801JUn5+vuLial4VNSYmRkFBQcrPz7/oNmfPnq3p06e7M3aDUlXt1Lj3t6m80qk+LRppRK8UsyMBAFBDrfesTJs27YJBs/85bdy4UZL05JNPql+/furYsaMefvhhLViwQH/6059UVFTk2t7FbopnGMYlb5Y3adIklZSUuKa8vLzafgv4gQVf7dPWvJOKCA7QnPs6yc+PmxQCADxLrfesZGVlafjw4ZddJzU19aLze/XqJUnau3evGjVqpPj4eK1bt67GOidOnFBlZeUFe1zOs1qtslqttY2Ni9hxuET//X/fS5Km39lOidEhJicCAOBCtS4rNptNNpvtmr7Yli1bJEkJCQmSpMzMTM2aNUtHjx51zVu6dKmsVqu6det2TV8DV6e8slrj39+mKqehwe3idXeXpmZHAgDgotw2ZmXNmjVau3at+vfvr6ioKG3YsEFPPvmk7rzzTiUnJ0uSBg4cqIyMDI0YMUJz5sxRcXGxJkyYoNGjR3MmkJv9dtke7S44JVt4kGbd3f6Sh90AADCb28qK1WrVe++9p+nTp8vhcCglJUWjR4/W008/7VrH399fS5Ys0WOPPaY+ffooJCREDzzwgObOneuuWJC0PqdYb3y9X5I0+56OahTOYTUAgOdyW1np2rWr1q5de8X1kpOT9fnnn7srBv7DaUeVxn+wVYYh3d+9mW7LuPjYIAAAPAVX/mpgZi3JVl7xGTWNDtFvfpRhdhwAAK6IstKArNhVoP9df/ZU77nDOikiONDkRAAAXBllpYE4UVqhZz7aLkka1TdNmc0bmZwIAICrQ1lpAAzD0JRPduj4KYdaNAnXU4Namx0JAICrRllpAD7bdkRLth9VgJ9Fv72/s4ID/c2OBADAVaOs+Lj8knL95pMdkqRf39JSHZpFmZwIAIDaoaz4MMMw9NSH22Qvr1KnZlF6rH9zsyMBAFBrlBUf9ta6g/r6+0JZA/w07/7OCvTnrxsA4H349PJRuUWlemHJd5KkZwa3UYsm4SYnAgDg2lBWfNSzi3boTGW1MtMb6aHeqWbHAQDgmlFWfNTWvJOKsAZo7v2d5OfHTQoBAN6LsuLDpt7ZTk2jQ8yOAQDAdaGs+JCKKqfr8YC2TXRv16YmpgEAoG5QVnzIy1/udj2eNjRDFguHfwAA3o+y4iM+//aI3ll30PW8UbjVxDQAANQdyooPyCks1cRzNykEAMDXUFa8XHlltR57e7NOO6rULSXG7DgAANQ5yoqXm754p747alejsCDNG9bR7DgAANQ5yooXW7T5kP53fZ4sFul/hndRk8hgsyMBAFDnKCte6vuCU5r88dm7KY8d0FJ9W9pMTgQAgHtQVrxQWUWVfvX2Zp2prFbfFjb9+paWZkcCAMBtKCtexjAMTfl4h/YeO60mEVb99ied5c/l9AEAPoyy4mXe35inRVsOy88izf9pFzWO4HoqAADfRlnxItlH7Hru052SpAmDWqtneiOTEwEA4H6UFS9xqrxSY97ZLEeVU/1bN9ajNzU3OxIAAPWCsuIFDMPQxI+2K6ewVIlRwXrl/s7yY5wKAKCBoKx4gb+tPaAl248qwM+iVx/sqpiwILMjAQBQbygrHu7bQyc18/NsSdLE29uoazKX1AcANCyUFQ9WUlapx97erMpqQwMz4jSqb5rZkQAAqHeUFQ9lGIYmfLhNh06cUVJsiOYM6ySLhXEqAICGh7Liof60KkfLsgsU5O+n1x7opqiQQLMjAQBgCsqKB9p0oFgvfrFLkvSboRnq0CzK5EQAAJiHsuJhiksrlPXOFlU5Df2oY4J+1jPZ7EgAAJjKrWUlNTVVFoulxjRx4sQa6/zncovFogULFrgzlsdyOg2Ne3+rjpaUK90Wphfv7cg4FQBAgxfg7i8wY8YMjR492vU8PDz8gnUWLlyowYMHu55HRTXMwx6vf7VP/9x9XNYAP/3+wa4Kt7r9rwcAAI/n9k/DiIgIxcfHX3ad6OjoK65T38oqqpTx3JeSpOwZgxQa5N63au3+Is1buluSNPPH7dU2IdKtXw8AAG/h9jErL730kho1aqTOnTtr1qxZqqiouGCdrKws2Ww29ejRQwsWLJDT6XR3LI9y/JRDv/7fLXIa0r1dm2lY92ZmRwIAwGO4dXfB2LFj1bVrV8XExGj9+vWaNGmScnJy9Oabb7rWmTlzpgYMGKCQkBAtX75c48ePV2FhoaZMmXLRbTocDjkcDtdzu93uzm/B7aqdhsa+u0XHTznUKi5cM+9qxzgVAAB+oNZlZdq0aZo+ffpl19mwYYO6d++uJ5980jWvY8eOiomJ0X333efa2yKpRinp3LmzpLPjXC5VVmbPnn3Fr+9N/mf591q9r0ihQf567cGubj/cBACAt6n1J2NWVpaGDx9+2XVSU1MvOr9Xr16SpL1797rKysXWsdvtKigoUFxc3AXLJ02apHHjxrme2+12JSUlXWV6z/L198c1f8X3kqRZd7dXiyYRJicCAMDz1Lqs2Gw22Wy2a/piW7ZskSQlJCRcdp3g4GBFR0dfdLnVapXVar2mr+9J8kvK9cS7W2UY0k9vSNbdXRinAgDAxbjtmMOaNWu0du1a9e/fX1FRUdqwYYOefPJJ3XnnnUpOPnuhs8WLFys/P1+ZmZkKCQnRypUrNXnyZD3yyCM+UUguparaqV//72YVlVYoIyFSU4dmmB0JAACP5bayYrVa9d5772n69OlyOBxKSUnR6NGj9fTTT7vWCQwM1GuvvaZx48bJ6XQqPT1dM2bM0JgxY9wVyyPMXbpHG3JPKNwaoNce7KrgQP862W5oUIByXxxSJ9sCAMBTuK2sdO3aVWvXrr3sOoMHD65xMbiGYPl3BVrw1T5J0sv3dVSqLczkRAAAeDbuDVSPDp0o07j3t0mSHuqdqjs6XHrsDgAAOIuyUk8qqpzKemeLSs5UqlOzKE26o43ZkQAA8AqUlXry4he7tDXvpCKDA/TqA11lDaibcSoAAPg6yko9+MeOo/rzNzmSpHn3d1ZSbKjJiQAA8B6UFTc7UFSqpz74VpL0Xzel67aMCy90BwAALo2y4kblldV67O3NOuWoUveUGE0Y1NrsSAAAeB3KihvN/DxbO4/YFRsWpPkPdFGgP283AAC1xaenm3y69bDeXndQFov0yv2dlBAVYnYkAAC8EmXFDfYeO61Ji7ZLkrL6t9DNrZuYnAgAAO9FWaljZyqqNebtzSqrqFZmeiM9cWsrsyMBAODVKCt17LlPd2h3wSnZwq36n592lr+fxexIAAB4NcpKHfpgY54+2HRIfhbpdz/trCYRwWZHAgDA61FW6sju/FP6zac7JElP3tpKvZvbTE4EAIBvoKzUgVJHlX719iaVVzp1U6vGGtO/hdmRAADwGZSV62QYhp79eLv2Hy9VfGSwfnt/J/kxTgUAgDpDWblO76w/qE+3HpG/n0WvPtBFjcKtZkcCAMCnUFauw47DJZr+WbYk6ZnBrdU9NdbkRAAA+B7KyjWyl1fqsbc3q6LaqVvbNtHoG9PNjgQAgE+irFwDwzD0zIff6mBxmZpGh2jesM6yWBinAgCAO1BWrsFfVufqix35CvS36PcPdlVUaKDZkQAA8FmUlVracvCEXvj7d5KkyXe0VeekaHMDAQDg4ygrtXCyrEJZ72xRZbWhOzrEa2TvVLMjAQDg8ygrV8npNDT+/W06fPKMUhuF6sV7OzJOBQCAekBZuUpvfL1fy3cdU1CAn37/YFdFBjNOBQCA+kBZuQqbDpzQnC93S5KmDW2ndolRJicCAKDhoKxchfHvb1O109BdnRP10xuSzI4DAECDQlm5CsdOOdS8cZhm3d2BcSoAANQzyspVCA700+s/66Ywa4DZUQAAaHAoK5ewZl+R6/FzP8pQq7gIE9MAANBwUVYuYU/BKdfju7o0NTEJAAANG2XlErjgGwAAnoGyAgAAPBplBQAAeDTKCgAA8GhuLytLlixRz549FRISIpvNpnvuuafG8oMHD2ro0KEKCwuTzWbT448/roqKCnfHAgAAXsKtFw756KOPNHr0aL3wwgu65ZZbZBiGtm/f7lpeXV2tIUOGqHHjxlq1apWKioo0cuRIGYah+fPnuzMaAADwEm4rK1VVVRo7dqzmzJmjUaNGuea3bt3a9Xjp0qXKzs5WXl6eEhMTJUnz5s3TQw89pFmzZikyMtJd8QAAgJdw22GgzZs36/Dhw/Lz81OXLl2UkJCg22+/XTt37nSts2bNGrVv395VVCRp0KBBcjgc2rRp00W363A4ZLfba0wAAMB3ua2s7N+/X5I0bdo0TZkyRZ9//rliYmLUr18/FRcXS5Ly8/MVFxdX43UxMTEKCgpSfn7+Rbc7e/ZsRUVFuaakJG4sCACAL6t1WZk2bZosFstlp40bN8rpdEqSJk+erHvvvVfdunXTwoULZbFY9MEHH7i2d7EbAxqGcckbBk6aNEklJSWuKS8vr7bfAgAA8CK1HrOSlZWl4cOHX3ad1NRUnTp19nL1GRkZrvlWq1Xp6ek6ePCgJCk+Pl7r1q2r8doTJ06osrLygj0uP9yG1WqtbWwAAOClal1WbDabbDbbFdfr1q2brFardu/erb59+0qSKisrlZubq5SUFElSZmamZs2apaNHjyohIUHS2UG3VqtV3bp1q200AADgg9x2NlBkZKQeffRRTZ06VUlJSUpJSdGcOXMkScOGDZMkDRw4UBkZGRoxYoTmzJmj4uJiTZgwQaNHj+ZMIAAAIMnN11mZM2eOAgICNGLECJ05c0Y9e/bUihUrFBMTI0ny9/fXkiVL9Nhjj6lPnz4KCQnRAw88oLlz57ozFgAA8CIWwzAMs0NcD7vdrqioKJWUlNTp3piyiiplPPelJCl7xiCFBrm11wEA0KDU5vObewMBAACPRlkBAAAejbICAAA8GmUFAAB4NMoKAADwaJQVAADg0SgrAADAo1FWAACAR6OsAAAAj0ZZAQAAHo2yAgAAPBplBQAAeDTKCgAA8GiUFQAA4NEoKwAAwKNRVgAAgEejrAAAAI9GWQEAAB6NsgIAADwaZQUAAHg0ygoAAPBolBUAAODRAswO4KlCgwKU++IQs2MAANDgsWcFAAB4NMoKAADwaJQVAADg0SgrAADAo1FWAACAR6OsAAAAj0ZZAQAAHo2yAgAAPBplBQAAeDTKCgAA8GiUFQAA4NHcXlaWLFminj17KiQkRDabTffcc0+N5RaL5YJpwYIF7o4FAAC8hFtvZPjRRx9p9OjReuGFF3TLLbfIMAxt3779gvUWLlyowYMHu55HRUW5MxYAAPAibisrVVVVGjt2rObMmaNRo0a55rdu3fqCdaOjoxUfH++uKAAAwIu57TDQ5s2bdfjwYfn5+alLly5KSEjQ7bffrp07d16wblZWlmw2m3r06KEFCxbI6XRecrsOh0N2u73GBAAAfJfb9qzs379fkjRt2jS98sorSk1N1bx589SvXz/t2bNHsbGxkqSZM2dqwIABCgkJ0fLlyzV+/HgVFhZqypQpF93u7NmzNX369AvmU1oAAPAe5z+3DcO48spGLU2dOtWQdNlpw4YNxttvv21IMv7whz+4XlteXm7YbDZjwYIFl9z+3LlzjcjIyEsuLy8vN0pKSlxTdnb2FfMwMTExMTExeeaUl5d3xe5R6z0rWVlZGj58+GXXSU1N1alTpyRJGRkZrvlWq1Xp6ek6ePDgJV/bq1cv2e12FRQUKC4u7oLlVqtVVqvV9Tw8PFx5eXmKiIiQxWKp7bdzWXa7XUlJScrLy1NkZGSdbhv/xvtcP3if6wfvc/3gfa4/7nqvDcPQqVOnlJiYeMV1a11WbDabbDbbFdfr1q2brFardu/erb59+0qSKisrlZubq5SUlEu+bsuWLQoODlZ0dPRV5fHz81OzZs2uat1rFRkZyQ9DPeB9rh+8z/WD97l+8D7XH3e811d79q/bxqxERkbq0Ucf1dSpU5WUlKSUlBTNmTNHkjRs2DBJ0uLFi5Wfn6/MzEyFhIRo5cqVmjx5sh555JEae08AAEDD5dbrrMyZM0cBAQEaMWKEzpw5o549e2rFihWKiYmRJAUGBuq1117TuHHj5HQ6lZ6erhkzZmjMmDHujAUAALyIW8tKYGCg5s6dq7lz5150+eDBg2tcDM7TWK1WTZ06lb08bsb7XD94n+sH73P94H2uP57wXlsM42rOGQIAADAHNzIEAAAejbICAAA8GmUFAAB4NMoKAADwaJSVS3jttdeUlpam4OBgdevWTV9//bXZkXzK7Nmz1aNHD0VERKhJkya66667tHv3brNj+bzZs2fLYrHoiSeeMDuKTzp8+LB+9rOfqVGjRgoNDVXnzp21adMms2P5lKqqKk2ZMkVpaWkKCQlxXfLicjfAxZX961//0tChQ5WYmCiLxaJPPvmkxnLDMDRt2jQlJiYqJCREN99880VvTOwulJWLeO+99/TEE09o8uTJ2rJli2688Ubdfvvtl71NAGrnq6++0pgxY7R27VotW7ZMVVVVGjhwoEpLS82O5rM2bNigN954Qx07djQ7ik86ceKE+vTpo8DAQH3xxRfKzs7WvHnzrvpq3Lg6L730khYsWKBXX31V3333nV5++WXNmTNH8+fPNzuaVystLVWnTp306quvXnT5yy+/rFdeeUWvvvqqNmzYoPj4eN12222uW+u43dXfwrDhuOGGG4xHH320xrw2bdoYEydONCmR7zt27Jghyfjqq6/MjuKTTp06ZbRs2dJYtmyZ0a9fP2Ps2LFmR/I5zzzzjNG3b1+zY/i8IUOGGL/85S9rzLvnnnuMn/3sZyYl8j2SjI8//tj13Ol0GvHx8caLL77omldeXm5ERUVd9sbEdYk9K/+hoqJCmzZt0sCBA2vMHzhwoFavXm1SKt9XUlIiSYqNjTU5iW8aM2aMhgwZoltvvdXsKD7rs88+U/fu3TVs2DA1adJEXbp00R//+EezY/mcvn37avny5dqzZ48kadu2bVq1apXuuOMOk5P5rpycHOXn59f4XLRarerXr1+9fS669Qq23qiwsFDV1dUX3PE5Li5O+fn5JqXybYZhaNy4cerbt6/at29vdhyf8+6772rz5s3asGGD2VF82v79+/X6669r3LhxevbZZ7V+/Xo9/vjjslqt+vnPf252PJ/xzDPPqKSkRG3atJG/v7+qq6s1a9Ys/fSnPzU7ms86/9l3sc/FAwcO1EsGysolWCyWGs8Nw7hgHupGVlaWvv32W61atcrsKD4nLy9PY8eO1dKlSxUcHGx2HJ/mdDrVvXt3vfDCC5KkLl26aOfOnXr99dcpK3Xovffe01tvvaV33nlH7dq109atW/XEE08oMTFRI0eONDueTzPzc5Gy8h9sNpv8/f0v2Ity7NixC1olrt+vf/1rffbZZ/rXv/6lZs2amR3H52zatEnHjh1Tt27dXPOqq6v1r3/9S6+++qocDof8/f1NTOg7EhISlJGRUWNe27Zt9dFHH5mUyDc99dRTmjhxooYPHy5J6tChgw4cOKDZs2dTVtwkPj5e0tk9LAkJCa759fm5yJiV/xAUFKRu3bpp2bJlNeYvW7ZMvXv3NimV7zEMQ1lZWVq0aJFWrFihtLQ0syP5pAEDBmj79u3aunWra+revbsefPBBbd26laJSh/r06XPB6fd79uxRSkqKSYl8U1lZmfz8an50+fv7c+qyG6WlpSk+Pr7G52JFRYW++uqrevtcZM/KRYwbN04jRoxQ9+7dlZmZqTfeeEMHDx7Uo48+anY0nzFmzBi98847+vTTTxUREeHakxUVFaWQkBCT0/mOiIiIC8YBhYWFqVGjRowPqmNPPvmkevfurRdeeEH333+/1q9frzfeeENvvPGG2dF8ytChQzVr1iwlJyerXbt22rJli1555RX98pe/NDuaVzt9+rT27t3rep6Tk6OtW7cqNjZWycnJeuKJJ/TCCy+oZcuWatmypV544QWFhobqgQceqJ+A9XLOkRf6/e9/b6SkpBhBQUFG165dOaW2jkm66LRw4UKzo/k8Tl12n8WLFxvt27c3rFar0aZNG+ONN94wO5LPsdvtxtixY43k5GQjODjYSE9PNyZPnmw4HA6zo3m1lStXXvR38siRIw3DOHv68tSpU434+HjDarUaN910k7F9+/Z6y2cxDMOon1oEAABQe4xZAQAAHo2yAgAAPBplBQAAeDTKCgAA8GiUFQAA4NEoKwAAwKNRVgAAgEejrAAAAI9GWQEAAB6NsgIAADwaZQUAAHg0ygoAAPBo/x8+KkfobWo07wAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "x_data = []\n", "y_data = []\n", "ydataerr = []\n", "times = [datetime.now()]\n", "\n", "def progress(it, metrics):\n", " times.append(datetime.now())\n", " x_data.append(it)\n", " y_data.append(metrics['eval/episode_reward'])\n", " ydataerr.append(metrics['eval/episode_reward_std'])\n", "\n", "# Each foot contacts the ground twice/sec.\n", "env = envs.get_environment(\"trotting_anymal\", step_k = 13)\n", "eval_env = envs.get_environment(\"trotting_anymal\", step_k = 13)\n", "\n", "make_inference_fn, params, _= train_fn(environment=env,\n", " progress_fn=progress,\n", " eval_env=eval_env)\n", "\n", "plt.errorbar(x_data, y_data, yerr=ydataerr)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "demo_env = envs.training.EpisodeWrapper(env, \n", " episode_length=1000, \n", " action_repeat=1)\n", "\n", "render_rollout(\n", " jax.jit(demo_env.reset),\n", " jax.jit(demo_env.step),\n", " jax.jit(make_inference_fn(params)),\n", " demo_env,\n", " n_steps=200\n", ")\n", "\n", "model_path = '/tmp/trotting_2hz_policy'\n", "model.save_params(model_path, params)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A note on sample efficiency**\n", "\n", "Above, we train using epochs * horizon_length * num_envs = 1.024e6 total simulator steps. Let's see what we get from PPO, using ten times more samples:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Text(0, 0.5, 'reward per episode')" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkcAAAGwCAYAAACjPMHLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABTpklEQVR4nO3deVhU1f8H8PewDfsgjLIIAuaCCCqKCpiKmkuaVqa5pVlqm6SFppGGQqllaZl9zcq11DTX3H5uuZcaIriAu2wKiKgwILLNnN8f5DQjqAwODMv79TzzNHPuwmduOPPm3HPulQghBIiIiIgIAGBk6AKIiIiIqhOGIyIiIiINDEdEREREGhiOiIiIiDQwHBERERFpYDgiIiIi0sBwRERERKTBxNAF1EQqlQqpqamwsbGBRCIxdDlERERUDkII5OTkwMXFBUZGj+4fYjiqgNTUVLi5uRm6DCIiIqqAlJQUuLq6PnI5w1EF2NjYACg5uLa2tgauhoiIiMpDoVDAzc1N/T3+KAxHFfDgVJqtrS3DERERUQ3zpCExdXZA9qJFi+Dp6Qlzc3O0a9cOR44cMXRJREREVA3UyXC0bt06fPDBB5g2bRpiYmLQuXNnPP/880hOTjZ0aURERGRgEiGEMHQRVa1jx45o27YtfvjhB3VbixYt8NJLL2HOnDlP3F6hUEAmkyE7O5un1YiIiGqI8n5/17meo8LCQkRHR6NXr15a7b169cLff/9d5jYFBQVQKBRaDyIiIqqd6lw4yszMhFKphKOjo1a7o6Mj0tPTy9xmzpw5kMlk6gen8RMREdVedS4cPfDwSHUhxCNHr4eFhSE7O1v9SElJqYoSiYiIyADq3FR+uVwOY2PjUr1EGRkZpXqTHpBKpZBKpVVRHhERERlYnes5MjMzQ7t27bB3716t9r179yIoKMhAVREREVF1Ued6jgAgNDQUI0eOhL+/PwIDA/HTTz8hOTkZ77zzjqFLIyIiIgOrk+FoyJAhuH37NiIjI5GWlgYfHx/s3LkT7u7uhi6NiIiIDKxOXufoafE6R0RERDUPr3NEREREVAEMR0REREQaGI6IiIiINDAcERERUbWQV1gMj493wOPjHcgrLDZYHXVythoREREZXn6REunZ+UhX5CM9Ox8pd/IMXRIAhiMiIiLSMyEEsvKKSkLPv8EnPTsfNxX5SPv3v+mKfGTlFT1yH/cLlbA0M0xMYTgiIiKq4fIKi+EdvhsAEB/Zu1JDRbFShYycAqQr8nEzWzvsqJ9n56OgWFWu/VmYGsNZZg5HW3PIbcyw7XQaAEDAcFcaYjgiIiIiAMC9gmJ16Hk47DzoAcrMLYCqnLnF3soMjrbm6vDj9OC57L82W3MT9Y3f8wqL1eHIUL1GAMMRERFRrSeEwJ17hVq9PA9OdaVrhJ+c/PINgjYxksDR1hyOtlI4yyxKgo9MCieZBZz+DUENbKUwNzXWqU5LMxMkftGvIm9RrxiOiIiIajilRlfOyr8TtYJQWnY+MhQFKFSW7zSXlZkxnGTmcPq3Z8dZVhJ2Sp5bwFEmhdxKCiMjSWW9HYNjOCIiIqphipQqnL2RjRPX7uBEwm2cTLyrXvblrouP3E5uXfo014Mg9KDNxty0Kt5CtcZwREREVM3lFylxOiULJxLu4J+EO4hOuov7Rcoy133exwkN7SzUoedBr4+jrTnMTHh5w/JgOCIiIqpm8gqLcSopCycSbuNEwh3EpmSh8KHZX/UsTdHB0x4dPB3Q2k2GQT8cAwDMe7W1QQcz1wY8ekREVGdV5RT4x1HkFyE68S6OJ9zGPwl3cPZ6NoofmhJW30aKDp72CPC0R8fGDmhS31o97seQV5OujRiOiIiIqtjde4X4J7HkFNmJhNuIT1WUmh7vIjNHx8YO6Ohpjw6e9vCUW6mnvFPlYjgiIiKqZBk5+fjn3/FCJ67dwcWbOaXW8XCwRAdPe3T0dEAHT3u42VsaoFICGI6IiIj0LjXrPk78e4rsxLU7uJZ5r9Q6TRpYq3uFOno6wElmXuGfV12uD1RbMBwRERE9BSEEku/k4cS/QehEwm1cv3tfax2JBPByskVHT3t09LRHe097yK2lBqqYnoThiIiISAdCCFy9lasOQ/8k3EG6Il9rHWMjCXxcbNGxsQM6eNijvYc9ZJa8flBNwXBERET0GCqVwIX0HPzz77T6fxLu4Pa9Qq11TI0laO1qV3KKrLED2rnXg7WUX7E1Ff/PERERaShWqhCXqlDPJPsn4Q4UD91zTGpihLaN6v0bhuzh51YPFma63UeMqi+GIyIiIgA/Hb6GmOQsRCfdRW6BdhiyMjNGOw979ZghX1cZpCYMQ7UVwxEREdVJKpXA2n+S1a+/3XdZ/dzG3AQdPEp6hTp4OsDHxRYmxrz1Rl3BcERERHXO+TQFPtl8FjHJWeq2nt4NEPSMHB087eHlZAvjWnzXeXo8hiMiIqoz8gqLseDPy1hyJAFKlYCV1Bj3Ckpu4LpgqB/vSUYAAPYREhFRnXDgYgZ6fXMYPx66BqVKoE9LJ2wLedbQZVE1xIhMRES1WoYiHxHb47HjTBoAoKGdBSIGtMRz3o68YSuVieGIiIh0Vl3uZv84KpXA6n+SMff/LiCnoBhGEmDMs5744LlmsOI1iOgx+NtBRES1TnxqyYDr2JQsAEBrVxlmvewLn4YywxZGNQLDERER1Rp5hcVYsO8ylhwtGXBtLTXBR72b47UAd84+o3JjOCIiolrhwIUMTN9yDjeySm76+ryPE2b0b/lUd7unuonhiIiIarSbinxEbovHjrOlB1wTVQTDERER1UhKlcCaE0mYu+sicgqKYWwkwZhnPTGxR1MOuKanwt8eIiKqccoacD17oC9auug24NrSzASJX/SrhAqpJmM4IiKiGiOvsBjf7ruMpRxwTZWI4YiIiGqE/Rdu4tMtceoB1319nRD+Agdck/4xHBERUbV2U5GPiG1x2Hk2HUDJgOvIF1uiRwsOuKbKwXBERETVklIlsPrfAde5GgOuP3iuabW8IjfVHrXmxrOJiYkYM2YMPD09YWFhgWeeeQYzZsxAYWGh1noSiaTUY/HixQaqmoiIyhKXmo2BP/yN8D/ikFtQjNZudtga0gmf9G3BYESVrtb8hl24cAEqlQo//vgjmjRpgnPnzmHcuHG4d+8evv76a611ly9fjj59+qhfy2S8nDwRUXVQ1oDrKX2aY0RHDrimqlNrwlGfPn20Ak/jxo1x8eJF/PDDD6XCkZ2dHZycnMq974KCAhQUFKhfKxSKpy+YiIi0/Hn+JsL/0B5wPaN/SzjacsA1Va1ac1qtLNnZ2bC3ty/VHhISArlcjvbt22Px4sVQqVSP3c+cOXMgk8nUDzc3t8oqmYiozknPzse7q6IxZuVJ3Mi6j4Z2Flg22h+LRrRjMCKDqDU9Rw+7evUqFi5ciHnz5mm1f/bZZ+jRowcsLCzw559/YtKkScjMzMT06dMfua+wsDCEhoaqXysUCgYkIqKnpFQJrDqehK92/zfgeuyznpjIAddkYNX+t2/mzJmIiIh47DpRUVHw9/dXv05NTUWfPn0wePBgjB07VmtdzRDUpk0bAEBkZORjw5FUKoVUKq1A9UREVJa41Gx8suksTl/PBgC0cbPD7Jd94e1ia+DKiGpAOAoJCcHQoUMfu46Hh4f6eWpqKrp164bAwED89NNPT9x/QEAAFAoFbt68CUdHXjODiKgy3Ssoxrf7LmHZX4lQqgRs/h1wPZwDrqkaqfbhSC6XQy6Xl2vdGzduoFu3bmjXrh2WL18OI6MnD6mKiYmBubk57OzsnrJSIiJ6nIcHXPfzdUZ4f2+OK6Jqp9qHo/JKTU1FcHAwGjVqhK+//hq3bt1SL3swM23btm1IT09HYGAgLCwscODAAUybNg1vvfUWT5sREVWS9OySK1z/37n/rnD92Ust0d2LvfVUPdWacLRnzx5cuXIFV65cgaurq9YyIQQAwNTUFIsWLUJoaChUKhUaN26MyMhIjB8/3hAlExHVakqVwK/HEvH1nksccE01ikQ8SA5UbgqFAjKZDNnZ2bC15eBBIqp78gqL4R2+GwAQH9m7VNg5dyMbn2w+izMccE3VSHm/vxndiYiq0JNCRU13r6AY3+y9hGV/JUAlUDLg+nkvDO/QiAOuqcaoXf8qiYjIYPbF30T4H+eQmp0PAOjXyhkzXvBGAw64phqG4YiIiJ5KenY+vtx1AbvjbgIoGXD9+Us+6ObVwMCVEVUMwxERET2VFxYeRV6hsmTAdWdPTOzBAddUs/G3l4iIdHYlI1f9PK9QiTZudpgz0BctnDngmmo+hiMiIio3IQR+OZaE2TvPq9vC+3tjdKAHjDjgmmoJhiMiIiqXjJx8fLT+DA5duqXVPrS9G4MR1SpPvr8GEZGB5RUWw+PjHfD4eAfyCosNXU6dtDf+Jvp8ewSHLt2CmYkRwvp6GbokokrDniMiInqkvMJifLb9PH77JxkA4OVkg++G+cG1ngXm7Lxg4OqIKgfDERERlenM9Sx8sDYW1zLvAQDGdfbE5N7NITUxZg8e1WoMR0REpEWpEvjh4BV8u+8yilUCTrbmmPdqa3RqIjd0aURVguGIiIjUUu7kIfT3WEQl3gUA9PV1wuyXfWFnaWbgyoiqDsMRERFBCIEtsTcQviUOOQXFsDIzRsSLPnilbUNIJJyJRnULwxERUR2Xfb8I07ecw7bTqQCAto3s8O0QPzRysDRwZUSGwXBERFSHHb92G6HrYpGanQ9jIwkmdG+K8d2egYkxr/RCdRfDERFRHVRYrML8vZfw4+GrEAJwd7DEN0PaoG2jeoYujcjgGI6IiOqYKxk5mLg2FnGpCgDAEH83fNrfG9ZSfiUQAQxHRER1hhACq44nYdbO88gvUsHO0hRfDPRFHx9nQ5dGVK0wHBER1QG3cgowdeMZ7L+QAQDo3FSOrwe3hqOtuYErI6p+GI6IiGq5P8/fxNSNZ5CZWwgzEyNM7eOFN4I8eLNYokdgOCIiqqXuFyoxa2c8Vh0vuS9ac0cbLBjWBl5OtgaujKh6YzgiIqqFzt3IxsS1Mbh6q+S+aGOe9cRHvZvD3NTYwJURVX8MR0REtYhSJfDT4WuYv/ciipQCDWykmPdqa3RuWt/QpRHVGAxHRES1xI2s+whdF4sTCXcAAH1aOmHOQF/Us+J90Yh0wXBERFQL/BF7A9O3nENOfjEszYwxs39LDPZ35X3RiCqA4YiIqAZT5BchfMs5bIktuS9aGzc7fDukDTzkVgaujKjmYjgiIqqh/km4gw/XxeJG1n0YSYD3uzdFSPcmMOV90YieCsMR6VVeYTG8w3cDAOIje8PSjL9iRPpWpFTh232X8MPBq1AJoJF9yX3R2rnzvmhE+sBvLiKiGuTqrVx8uC4WZ65nAwAGtXPFzAEteV80Ij3ivyaiWow9ebWHEAJr/knG59vP436REjILU8wZ6Iu+vrwvGpG+8ZOSiKiau51bgKkbz2Lf+ZsAgE5NHPD14NZwllkYuDKi2onhiIioGjtwMQMfrT+DzNwCmBkbYUqf5nizkyfvi0ZUiRiOiIiqofwiJebsPI+Vx5IAAM0crfHtED94u/C+aESVjeGIqAwcq0OGFJeajQ/WxuJyRi4AYHSQBz5+3qta3RfN0swEiV/0M3QZRJWCn/hERNWESiWw5Og1fLW75L5o9W2k+Hpwa3RtxvuiEVUlhiMiomogNes+Jv1+Gseu3QYA9PJ2xBevtII974tGVOUYjoiIDGz7mVR8suksFPnFsDA1xoz+3hjS3o33RSMykFp1jXkPDw9IJBKtx8cff6y1TnJyMvr37w8rKyvI5XJMmDABhYWFBqqYiOqy3PxihP4ei5A1MVDkF6O1qww7J3bG0A6NGIyIDKjW9RxFRkZi3Lhx6tfW1tbq50qlEv369UP9+vVx9OhR3L59G6+//jqEEFi4cKEhyiWiOmzgD3/j+t2S+6KN79YEE3o05X3RiKqBWheObGxs4OTkVOayPXv2ID4+HikpKXBxcQEAzJs3D6NHj8asWbNga8spskRUuYqUKvXz63fvw7WeBb4d0gb+HvYGrIqINNW6P1G+/PJLODg4oE2bNpg1a5bWKbNjx47Bx8dHHYwAoHfv3igoKEB0dPQj91lQUACFQqH1ICLS1fFrtzFw0d/q1y+2ccH/TezMYERUzdSqnqOJEyeibdu2qFevHv755x+EhYUhISEBS5YsAQCkp6fD0dFRa5t69erBzMwM6enpj9zvnDlzEBERUam1E1HtlZGTjzk7L2BzzA2t9jkDfXkNLaJqqNr3HM2cObPUIOuHHydPngQAfPjhh+jatStatWqFsWPHYvHixVi6dClu376t3l9ZgxyFEI8d/BgWFobs7Gz1IyUlRf9vlIhqHaVKYOXfiegx7xA2x9yARAIMbe9m6LKI6Amq/Z8sISEhGDp06GPX8fDwKLM9ICAAAHDlyhU4ODjAyckJJ06c0Frn7t27KCoqKtWjpEkqlUIqlepWOBHVaTHJd/HpH+dw7kbJafhWrjJ89qIPmjpaY20U/8Aiqs6qfTiSy+WQy+UV2jYmJgYA4OzsDAAIDAzErFmzkJaWpm7bs2cPpFIp2rVrp5+CiahOy8orxJe7LmJtVDKEAGzNTfBRHy8M79AIxkYS5BUWG7pEInqCah+OyuvYsWM4fvw4unXrBplMhqioKHz44YcYMGAAGjVqBADo1asXvL29MXLkSHz11Ve4c+cOJk+ejHHjxnGmGhE9FZVKYEP0dXyx6wLu3CuZCPJKW1eE9fWC3Jo9z0Q1Sa0JR1KpFOvWrUNERAQKCgrg7u6OcePGYcqUKep1jI2NsWPHDrz33nvo1KkTLCwsMHz4cHz99dcGrJyIarr4VAU+/eMcopPuAgCaO9rgs5d80MGTs9CIaqJaE47atm2L48ePP3G9Ro0aYfv27VVQERHVdjn5Rfhm72WsPJYIpUrAyswYHzzXDKM7efBijkQ1WK0JR0REVUUIgW1n0vD59nhk5BQAAPr5OmP6Cy3gLLMwcHVE9LQYjoiIdHD1Vi7C/ziHv66UXCLEw8ESkS/6oEuz+gaujIj0heGIiKgc7hcq8f2By/jp8DUUKQWkJkYY360J3urSGOamxoYuj4j0iOGIiOgJ9sbfxMytcbiRdR8A0K15fUQM8EEjB0sDV0ZElYHhiIjoEVLu5CFiWxz2nc8AALjIzDFjQEv08nZ87FX1iahmYziqJvIKi+EdvhsAEB/Zm/dbIjKggmIlfj58DQv3X0FBsQqmxhKM7dwY73dvwn+bRHUA/5UTEWk4ejkT4X+cw7XMewCAwMYO+OyllmjSwMbAlRFRVWE4IiICkJ6dj893xGP7mTQAQH0bKab3a4EBrV14Co2ojmE4IqI6rVipwoq/E/HN3ku4V6iEkQQYFeiB0F7NYGtuaujyiMgAKhSOjhw5gh9//BFXr17Fhg0b0LBhQ/z666/w9PTEs88+q+8aiYgqxcnEO5i+5RwupOcAAPwa2eGzF33g01Bm4MqIyJB0vr79xo0b0bt3b1hYWCAmJgYFBSVXh83JycHs2bP1XiARkb7dzi3A5PWnMWjxMVxIz4GdpSm+GOiLje8EMRgRke7h6PPPP8fixYvx888/w9T0vy7noKAgnDp1Sq/FERHpk1IlsPpEErrPO4QN0dcBAEPbu2H/pGAM7dAIRkYcW0REFTitdvHiRXTp0qVUu62tLbKysvRRExGR3p29no3pW87i9PVsAIC3sy0+e8kH7dzrGbgyIqpudA5Hzs7OuHLlCjw8PLTajx49isaNG+urLiIivci+X4R5ey7i1+NJEAKwkZogtFczjAxwh4mxzp3nRFQH6ByO3n77bUycOBHLli2DRCJBamoqjh07hsmTJyM8PLwyaiQi0pkQAptjbmD2zvPIzC0EALzYxgXT+rZAA1tzA1dHRNWZzuFoypQpyM7ORrdu3ZCfn48uXbpAKpVi8uTJCAkJqYwaiYh0cjE9B5/+cQ7/JNwBADxT3wqfveiDoCZyA1dGRDVBhabyz5o1C9OmTUN8fDxUKhW8vb1hbW2t79qIiHRyr6AYC/68jGVHE1CsEjA3NcKEHk0x9tnGMDPhKTQiKp8KXwTS0tIS/v7++qyFiKhChBDYdS4dkdvjkZadDwDo5e2I8P7ecK1naeDqiKimKVc4GjhwYLl3uGnTpgoXQ0Skq8TMewjfGofDl24BAFzrWSBiQEv0aOFo4MqIqKYqVziSyf67KJoQAps3b4ZMJlP3HEVHRyMrK0unEEVE9DTyi5RYdPAqFh+6isJiFcyMjfBO18Z4r1sTmJsaG7o8IqrByhWOli9frn4+depUvPrqq1i8eDGMjUs+gJRKJd577z3Y2tpWTpVERBoOXMjAjK1xSL6TBwDo3FSOyBd94Cm3MnBlRFQb6DzmaNmyZTh69Kg6GAGAsbExQkNDERQUhK+++kqvBRIRaZrwWwz2nc8AADjZmuPTF7zR19cJEgmvbk1E+qFzOCouLsb58+fRvHlzrfbz589DpVLprTAiogcKipTq5/vOZ8DYSII3O3lg4nPNYC2t8LwSIqIy6fyp8sYbb+DNN9/ElStXEBAQAAA4fvw4vvjiC7zxxht6L5CI6i4hBPadz0Dktjh1W9tGdpg90BdeTjyNT0SVQ+dw9PXXX8PJyQnffPMN0tLSAJTcUmTKlCmYNGmS3gskorrpSkYOIrbF48jlTK32X97sAGtz00dsRUT09HQOR0ZGRpgyZQqmTJkChUIBAByITUR6o8gvwoJ9l7Hy70QUqwTMjI0wKsgdS44kAACMjDi2iIgqV4VP1t+6dQsXL16ERCJB8+bNIZfzsvxEVHEqlcD66BTM3XURt++V3AvtuRYNML2fNxrYStXhiIiosukcju7du4f3338fv/zyi3oAtrGxMUaNGoWFCxfC0pJXoyUi3UQn3UXEtjicuZ4NAGhc3wrhL3gjuHkDAEBeYbEhyyOiOkbnmw2Fhobi0KFD2LZtG7KyspCVlYU//vgDhw4d4pgjItLJTUU+QtfF4pUf/saZ69mwlppger8W2DWxizoYERFVNZ17jjZu3IgNGzYgODhY3da3b19YWFjg1VdfxQ8//KDP+oioFiooVmLZ0UQs3H8ZeYUl0/Rf9XfFR729UN9GauDqiKiu0zkc5eXlwdGx9D2LGjRogLy8PL0URUS1kxAC+y9k4LPt8Ui8XfJ54dfIDjP7t0RrNzvDFkdE9C+dw1FgYCBmzJiBX375Bebm5gCA+/fvIyIiAoGBgXovkIhqh6u3chG5LR6H/r1BbH0bKT7u44WX/RpyBhoRVSs6h6MFCxagT58+cHV1RevWrSGRSBAbGwtzc3Ps3r27MmokohosJ78I3/15Gcv/Kpmab2oswZvPeuL97k15dWsiqpZ0/mTy8fHB5cuXsWrVKly4cAFCCAwdOhQjRoyAhYVFZdRIRDWQSiWw4dR1zN11EZm5BQCA7l4N8OkL3nX6BrGWZiZI/KKfocsgoseo0J9tFhYWGDdunL5rIaJaIib5LmZui8fplCwAgKe8ZGp+Ny/OQCOi6k/nqfwrV67Ejh071K+nTJkCOzs7BAUFISkpSa/FEVHNkqHIx6TfT+PlRX/jdEoWrKUm+KSvF3Z/0IXBiIhqDJ3D0ezZs9Wnz44dO4bvv/8ec+fOhVwux4cffqj3Aomo+issVuHHQ1fR7euD2HjqOgBgUDtX7J/cFW91eQZmJjp/1BARGYzOn1gpKSlo0qQJAGDLli0YNGgQ3nrrLcyZMwdHjhzRe4HldfDgQUgkkjIfUVFR6vXKWr548WKD1U1U0x24kIHe3x7GnP+7gHuFSrR2s8Pm94Lw9eDWaGBjbujyiIh0pvOYI2tra9y+fRuNGjXCnj171L1F5ubmuH//vt4LLK+goCCkpaVptX366afYt28f/P39tdqXL1+OPn36qF/LZLIqqZGoNrl2KxefbY/HgYslU/Pl1lJM7dMcr7R15dR8IqrRdA5HPXv2xNixY+Hn54dLly6hX7+SWRdxcXHw8PDQd33lZmZmBicnJ/XroqIibN26FSEhIZBItD+o7ezstNYlovLLLSjGwv2XsexoAoqUJVPz3+jkife7N4GNuamhyyMiemo6n1b73//+h8DAQNy6dQsbN26Eg4MDACA6OhrDhg3Te4EVtXXrVmRmZmL06NGlloWEhEAul6N9+/ZYvHix+ga6j1JQUACFQqH1IKprVCqBDdHX0e3rg/jx0DUUKQWCm9fHrg+64JO+LRiMiKjW0LnnyM7ODt9//32p9oiICL0UpC9Lly5F79694ebmptX+2WefoUePHrCwsMCff/6JSZMmITMzE9OnT3/kvubMmVPt3h9RVTqdkoUZW+MQ++/UfA8HS4T390Z3r9K3EiIiqunKFY7OnDkDHx8fGBkZ4cyZM49dt1WrVnop7IGZM2c+MZhERUVpjSu6fv06du/ejd9//73UupohqE2bNgCAyMjIx4ajsLAwhIaGql8rFIpSoYuoNrqVU4C5uy5gfXTJDDQrM2O836Mp3ujkAamJsYGrIyKqHOUKR23atEF6ejoaNGiANm3aQCKRQAihXv7gtUQigVKp1GuBISEhGDp06GPXeXis0/Lly+Hg4IABAwY8cf8BAQFQKBS4efNmmTfUBQCpVAqplHcKp7qjsFiFlX8n4rs/LyOnoBgAMLBtQ0zt4wVHW85AI6LarVzhKCEhAfXr11c/r0pyuRxyubzc6wshsHz5cowaNQqmpk8eAxETEwNzc3PY2dk9RZVEtcfBixmI3B6Pa7fuAQBaucowc0BLtG1Uz8CVERFVjXKFI3d39zKfV0f79+9HQkICxowZU2rZtm3bkJ6ejsDAQFhYWODAgQOYNm0a3nrrLfYMUZ2XmHkPn++Ix77zGQAAubUZpvT2wqB2nJpPRHVLhe6tdvHiRSxcuBDnz5+HRCKBl5cX3n//fTRv3lzf9els6dKlCAoKQosWLUotMzU1xaJFixAaGgqVSoXGjRsjMjIS48ePN0ClRNVDbkExvt9/BcuOJqBQqYKJkQSjgzww4bmmsOUMNCKqg3QORxs2bMCwYcPg7++PwMBAAMDx48fh4+ODNWvWYPDgwXovUhdr1qx55LI+ffpoXfyRqC4TQmBL7A3M2XkBGTkFAIAuzeoj/AVvNGlgbeDqiIgMR+dwNGXKFISFhSEyMlKrfcaMGZg6darBwxERPdmZ61mYuTUOp5KzAADuDpb4tJ83erRoUOqiqUREdY3O4Sg9PR2jRo0q1f7aa6/hq6++0ktRRFQ5MnML8NWui/g9OgVCAJZmxgjp3gRjnvXk1Hwion/pHI6Cg4Nx5MgR9c1nHzh69Cg6d+6st8KISL9W/p2IRQeuqqfmv+xXMjXfScap+UREmnQORwMGDMDUqVMRHR2NgIAAACVjjtavX4+IiAhs3bpVa10iMpz8ov+uO/blrosAAN+GMswc4I127vaGKouIqFrTORy99957AIBFixZh0aJFZS4DUCkXhCSi8olLzca6qBRsibmhbrO3MsOU3s0x2N8NxpyaT0T0SDqHoyfdpJWIDEORX4Q/YlPxe1QKzt7ILrV854Rn4SSzMEBlREQ1S4Wuc/RAfn4+zM05XoHIUIQQiEq8i7VRydh5Ng35RSV/vJgZG6FnS0e83MYFY3+JBgDYWvCaRURE5aFzOFIqlZg9ezYWL16Mmzdv4tKlS2jcuDE+/fRTeHh4lHllaiLSr4ycfGw6dQO/R6XgWuY9dXszR2sMad8IL/s1hL2VGfIKiw1YJRFRzaRzOJo1axZWrlyJuXPnYty4cep2X19ffPPNNwxHRJWkWKnC4cu3sPafFPx5IQNKVcnNn63MjNG/tQuGtHdDGzc7XqeIiOgp6RyOfvnlF/z000/o0aMH3nnnHXV7q1atcOHCBb0WR0RA8u08/H4yBeujU3BTUaBub9vIDkPbN0K/Vs6wkj7VGXIiItKg8yfqjRs3Sl3jCCgZqF1UVKSXoojquvwiJXbHpWNdVAr+vnpb3W5vZYaBfg0xpL0bmjraGLBCIqLaS+dw1LJlSxw5cgTu7u5a7evXr4efn5/eCiOqi86nKbAuKgWbY24g+37JHxsSCdC5aX0M8XfDc94NeCVrIqJKpnM4mjFjBkaOHIkbN25ApVJh06ZNuHjxIn755Rds3769MmokqtVy8ouw9XTJFPzT1/+bgt/QzgKD/V0xqJ0rXOtZGrBCIqK6Redw1L9/f6xbtw6zZ8+GRCJBeHg42rZti23btqFnz56VUSNRrSOEwMmku1gXlYIdZ9Jw/98rWZsaS9DL2wmvtnfDs03kvFgjEZEBVGgUZ+/evdG7d29910JU62XmFmDTqetYG5WCa7f+m4LfpIE1hrZ3w8t+DeFgLTVghURExCkuRJVMqRI4fPkW1v2Tgn3nb6L43yn4lmbGeKGVM4a0b4S2jTgFn4ioumA4IqokKXfysP5kCtZHX0dadr66vY2bHYa2d8MLrV1gzSn4RETVDj+ZifSooFiJPXE3sS4qBUevZKrb7SxNMdDPFUPau6G5E6fgExFVZwxHRHpwIf2/KfhZef9d76tzUzmGtHdDT29HTsEnIqohdApHRUVFaN68ObZv3w5vb+/KqomoRsgtKMa206lYG5WC0ylZ6nZnmTkG+7thcDtXuNlzCj4RUU2jUzgyNTVFQUEBB45SnSWEwKnkkin428+kIa+wZAq+iZEEPb0dMaS9Gzo3rc8p+ERENZjOp9Xef/99fPnll1iyZAlMTHhWjuqG27kF2HTqBtadTMGVjFx1+zP1rTCkvRsGtnWFnFPwiYhqBZ3TzYkTJ/Dnn39iz5498PX1hZWVldbyTZs26a04ourgg3WxOHAhA0XKkin4FqYPpuC7oZ17PfakEhHVMjqHIzs7O7zyyiuVUQtRtXArpwCrTySpX++JuwkAaO1mhyH+bujf2hk25qaGKo+IiCqZzuFo+fLllVEHkUEJIXAi4Q5WHU/C7rh0dS8RALwW0AgjOrqjhbOtASskIqKqUqFBQ8XFxTh48CCuXr2K4cOHw8bGBqmpqbC1tYW1tbW+aySqNIr8ImyKvo7VJ5JxWWMsUWtXmfomsJ/0bQFLM46vIyKqK3T+xE9KSkKfPn2QnJyMgoIC9OzZEzY2Npg7dy7y8/OxePHiyqiTSK/O3cjGquNJ+CM2VX3TV0szY7zYpiFGdGyExvWt4B2+28BVEhGRIegcjiZOnAh/f3+cPn0aDg4O6vaXX34ZY8eO1WtxRPp0v1CJbWdSsfp4krpXCACaOVrjtQB3vOTXELb/jiXKKyw2VJlERGRgOoejo0eP4q+//oKZmZlWu7u7O27cuKG3woj05eqtXKw+nowN0SlQ5JeEHjNjIzzv64QRHd3R3oMzzoiI6D86hyOVSgWlUlmq/fr167Cx4T2jqHooUqqwN/4mVh1Pwt9Xb6vb3ewtMLyDOwb787pERERUNp3DUc+ePfHtt9/ip59+AgBIJBLk5uZixowZ6Nu3r94LJNJFatZ9rP0nGWujUpCRUwAAMJIA3b0aYESAO7o2rQ8jXr2aiIgeQ+dw9M0336Bbt27w9vZGfn4+hg8fjsuXL0Mul+O3336rjBqJHkulEjh8+RZWHU/G/gs3ofp3Fn59GymGtnfD0A6N0NDOwrBFEhFRjaFzOHJxcUFsbCx+++03nDp1CiqVCmPGjMGIESNgYcEvIKo6t3MLsD76OtacSEbynTx1e2BjB7wW4I5eLR1hamxkwAqJiKgmqtDFWywsLPDmm2/izTff1Hc9RI8lhMDJpLtYfTwJO8+mo1CpAgDYmJtgUDtXjOjojiYNeK0tIiKquAqFo4sXL2LhwoU4f/48JBIJvLy8EBISAi8vL33XRwQAyMkvwpaYG1h9IhkX0nPU7a1cZXitozv6t3aBhZmxASskIqLaQudwtGHDBgwbNgz+/v4IDAwEABw/fhy+vr5Ys2YNBg8erPciqe6KT1Vg1Ykk/BFzA/cKS2ZJmpsa4cXWDTEioBFaudoZtkAiIqp1dA5HU6ZMQVhYGCIjI7XaZ8yYgalTpzIc0VPLL1Ji59k0rDqehFPJWer2Z+pb4bUAdwxs6wqZBW/8SkRElUPn0arp6ekYNWpUqfbXXnsN6enpeimqLLNmzUJQUBAsLS1hZ2dX5jrJycno378/rKysIJfLMWHCBBQWFmqtc/bsWXTt2hUWFhZo2LAhIiMjIYQoc39UtRIz72HWjngEzPkTob+fxqnkLJgYSdCvlTN+GxeAfaFd8UYnTwYjIiKqVDr3HAUHB+PIkSNo0qSJVvvRo0fRuXNnvRX2sMLCQgwePBiBgYFYunRpqeVKpRL9+vVD/fr1cfToUdy+fRuvv/46hBBYuHAhAEChUKBnz57o1q0boqKicOnSJYwePRpWVlaYNGlSpdVOj1asVGHf+QysPpGEI5cz1e0N7SwwvGMjDPZ3RQMbcwNWSEREdY3O4WjAgAGYOnUqoqOjERAQAKBkzNH69esRERGBrVu3aq2rLxEREQCAFStWlLl8z549iI+PR0pKClxcXAAA8+bNw+jRozFr1izY2tpi9erVyM/Px4oVKyCVSuHj44NLly5h/vz5CA0N5S0kqlB6dj7WRiVj7T8pSFfkAwAkEiC4WX28FuCO4OYNYMyLNdK/LM1MkPhFP0OXQUR1hM7h6L333gMALFq0CIsWLSpzGVBy5eyybjNSWY4dOwYfHx91MAKA3r17o6CgANHR0ejWrRuOHTuGrl27QiqVaq0TFhaGxMREeHp6lrnvgoICFBQUqF8rFIrKeyO1mEol8NfVTKw6noR95zOg/PdqjQ5WZni1vRuGd2gEN3tLA1dJRER1XYXurVYdpaenw9HRUautXr16MDMzU4+FSk9Ph4eHh9Y6D7ZJT09/ZDiaM2eOuueKdHf3XiE2RF/H6hNJSLz938UaO3ja47UAd/Ru6QipCafhExFR9VCh6xzpy8yZM58YOqKiouDv71+u/ZV1WkwIodX+8DoPBmM/7pRaWFgYQkND1a8VCgXc3NzKVVNddjolC+ujr2P7mTQUFv97sUapCQa2bYgRAe5o5sgbFRMRUfVj0HAUEhKCoUOHPnadh3t6HsXJyQknTpzQart79y6KiorUvUNOTk6lZtRlZGQAQKleJ01SqVTrVBw9mubMv2E///f/o6WLLV4LcMeA1i6wkhr0146IiOixDPotJZfLIZfL9bKvwMBAzJo1C2lpaXB2dgZQMkhbKpWiXbt26nU++eQTFBYWwszMTL2Oi4tLuUMYPdq1W7kI23RW/VpqYoQXWrngtYBGaONmxwHvRERUI9SYP+GTk5Nx584dJCcnQ6lUIjY2FgDQpEkTWFtbo1evXvD29sbIkSPx1Vdf4c6dO5g8eTLGjRsHW1tbAMDw4cMRERGB0aNH45NPPsHly5cxe/ZshIeH84v7KeQXKfHDwav44eBV9b3OAODA5K5wseMAayIiqllqTDgKDw/HypUr1a/9/PwAAAcOHEBwcDCMjY2xY8cOvPfee+jUqRMsLCwwfPhwfP311+ptZDIZ9u7di/Hjx8Pf3x/16tVDaGio1ngi0s3fVzIxbcs5JGTeAwA820SOo1dKrldkZ2lmyNKIiIgqpFzhSJep6w96afRtxYoVj7zG0QONGjXC9u3bH7uOr68vDh8+rMfK6qbM3ALM2nEem2NuAAAa2Egxo39LBDeXo+WMPQaujoiIqOLKFY7s7Mo/XqQqr21EVU+lElgblYIv/u88FPnFkEiAUQHumNS7OWzNTZFXWGzoEomIiJ5KucLRgQMH1M8TExPx8ccfY/To0QgMDARQcgHGlStXYs6cOZVTJVULF9IVmLb5HKKT7gIomYE2+2VftHazM2xhREREelSucNS1a1f188jISMyfPx/Dhg1Ttw0YMAC+vr746aef8Prrr+u/SjKovMJiLPjzMpYeSUCxSsDKzBiTejXHqEB3mBjrfO9iIiKiak3nAdnHjh3D4sWLS7X7+/tj7NixeimKqo8/z99E+B9xuJF1HwDQp6UTZgzwhrPMwsCVERERVQ6d/+x3c3MrMxz9+OOPvGp0LZKWfR/v/BqNMStP4kbWfTS0s8DS1/2xeGQ7BiMiIqrVdO45+uabb/DKK69g9+7dCAgIAAAcP34cV69excaNG/VeIFWtYqUKvxxLwrw9F3GvUAljIwnGdvbExB5NYWlWY678QEREVGE6f9v17dsXly9fxg8//IDz589DCIEXX3wR77zzDnuOarjTKVmYtuUszt0ouXRD20Z2mD3QF15OlXN5BiIioupIp3BUVFSEXr164ccff8SsWbMqqyaqYor8IszbfRG/HE+CEICtuQk+fr4FhrZ3g5ERrxxORER1i07hyNTUFOfOneOtNmoJIQR2nk1HxLY4ZOQUAABe9muIT/q2QH0b3miXiIjqJp0HZI8aNQpLly6tjFqoCiXfzsMbK6Iwfs0pZOQUwFNuhVVjOuKbIW0YjIiIqE7TecxRYWEhlixZgr1798Lf3x9WVlZay+fPn6+34kj/CotV+PnINXz352UUFKtgZmyEd4OfwbvBz8Dc1NjQ5RERERmczuHo3LlzaNu2LQDg0qVLWst4uq16i0q8g2mbz+LSzVwAQNAzDvjsJR88U9/awJURERFVHzqHI81biVDNcPdeIb74vwtYdzIFAOBgZYZp/VrgZb+GDLREREQP4YVrajEhBDaeuoHZO8/jzr1CAMCwDm6Y2scLdpZmBq6OiIioeqpQOIqKisL69euRnJyMwsJCrWWbNm3SS2H0dK5k5GL6lrM4fu0OAKC5ow1mvewDfw97A1dGRERUvek8W23t2rXo1KkT4uPjsXnzZhQVFSE+Ph779++HTCarjBpJB/lFSszfcxHPLziM49fuwNzUCFP7eGH7hGcZjIiIiMpB556j2bNn45tvvsH48eNhY2ODBQsWwNPTE2+//TacnZ0ro0YqpyOXb2H6lnNIup0HAOjWvD4iX/SBm72lgSsjIiKqOXQOR1evXkW/fv0AAFKpFPfu3YNEIsGHH36I7t27IyIiQu9F0uNl5OTj8+3nsfV0KgDA0VaKmf1boo+PEwdcExER6UjncGRvb4+cnBwAQMOGDXHu3Dn4+voiKysLeXl5ei+QHk2lEljzTzK+3HUBOfnFMJIAowI9MKlXM9iYmxq6PCIiohpJ53DUuXNn7N27F76+vnj11VcxceJE7N+/H3v37kWPHj0qo0YqQ3yqAp9sPovYlCwAgG9DGWa/7AtfV477IiIieho6h6Pvv/8e+fn5AICwsDCYmpri6NGjGDhwID799FO9F0ja7hUU49t9l7Dsr0QoVQLWUhNM7tUMIwM9YMybxBIRET21Cp1We8DIyAhTpkzBlClT9FoUlW1v/E3M+OMcUrNLwmlfXyeEv9ASTjJzA1dGRERUe+gcjkaMGIHg4GB07doVzZo1q4ya6CGpWfcxY2sc9sbfBAC41rPAZy/6oJtXAwNXRkREVPvoHI6sra0xb948vP3223ByckLXrl3RtWtXBAcHw8vLqzJqrLOKlSqs+DsR8/deQl6hEiZGEozr0hgTujeFhRlvEktERFQZdA5HP/74IwAgPT0dBw8exMGDB7FgwQKMHz8eDRo0QFpamt6LrItiU7LwyaaziE9TAAD83eth1su+aO5kY+DKiIiIarcK31vNxsYG9erVQ7169WBnZwcTExM4OTnps7Y667Pt8VgblQIhAJmFKcKe98Kr/m4w4oBrIiKiSqdzOJo6dSoOHTqE06dPw8fHB126dEFYWBi6dOkCOzu7SiixbhBCqJ//9k8KAGBg24aY1rcFHKylhiqLiIioztE5HH311VeoX78+ZsyYgRdffBEtWrSojLrqnKQ7/11A08PBErMH+iLoGbkBK6LawNLMBIlf9DN0GURENYrO4SgmJgaHDh3CwYMHMW/ePBgbG6sHZAcHBzMsVZCHg5X6+ZbxnWBnaWbAaoiIiOouncNR69at0bp1a0yYMAEAcPr0aXz77beYMGECVCoVlEql3ousa8xMjAxdAhERUZ1VoQHZMTEx6plqR44cgUKhQJs2bdCtWzd910dERERUpXQOR/Xq1UNubi5at26N4OBgjBs3Dl26dIGtrW1l1EdERERUpXQOR7/++ivDEBEREdVaOg9ueeGFF2Bra4srV65g9+7duH//PgDtqehERERENZXO4ej27dvo0aMHmjVrhr59+6qviD127FhMmjRJ7wUSERERVSWdw9GHH34IU1NTJCcnw9LSUt0+ZMgQ7Nq1S6/FEREREVU1nccc7dmzB7t374arq6tWe9OmTZGUlKS3woiIiIgMQeeeo3v37mn1GD2QmZkJqbTybnMxa9YsBAUFwdLSsszblJw+fRrDhg2Dm5sbLCws0KJFCyxYsEBrncTEREgkklIP9ngRERHRAzqHoy5duuCXX35Rv5ZIJFCpVPjqq68q9TpHhYWFGDx4MN59990yl0dHR6N+/fpYtWoV4uLiMG3aNISFheH7778vte6+ffuQlpamfnTv3r3S6iYiIqKapUL3VgsODsbJkydRWFiIKVOmIC4uDnfu3MFff/1VGTUCACIiIgAAK1asKHP5m2++qfW6cePGOHbsGDZt2oSQkBCtZQ4ODnBycqqUOomIiKhm07nnyNvbG2fOnEGHDh3Qs2dP3Lt3DwMHDkRMTAyeeeaZyqixwrKzs2Fvb1+qfcCAAWjQoAE6deqEDRs2PHE/BQUFUCgUWg8iIiKqnXTqOSoqKkKvXr3w448/qntyqqtjx47h999/x44dO9Rt1tbWmD9/Pjp16gQjIyNs3boVQ4YMwcqVK/Haa689cl9z5syp9u+XiIiI9EOnniNTU1OcO3cOEolELz985syZZQ6Q1nycPHlS5/3GxcXhxRdfRHh4OHr27Klul8vl+PDDD9GhQwf4+/sjMjIS7733HubOnfvY/YWFhSE7O1v9SElJ0bkmIiIiqhl0HnM0atQoLF26FF988cVT//CQkBAMHTr0set4eHjotM/4+Hh0794d48aNw/Tp05+4fkBAAJYsWfLYdaRSaaXOxCMiIqLqQ+dwVFhYiCVLlmDv3r3w9/eHlZWV1vL58+eXe19yuRxyuVzXEh4pLi4O3bt3x+uvv45Zs2aVa5uYmBg4OzvrrQYiIiKq2XQOR+fOnUPbtm0BAJcuXdJapq/TbWVJTk7GnTt3kJycDKVSidjYWABAkyZNYG1tjbi4OHTr1g29evVCaGgo0tPTAQDGxsaoX78+AGDlypUwNTWFn58fjIyMsG3bNnz33Xf48ssvK61uIiIiqll0DkcHDhyojDqeKDw8HCtXrlS/9vPzU9cTHByM9evX49atW1i9ejVWr16tXs/d3R2JiYnq159//jmSkpJgbGyMZs2aYdmyZY8djE1ERER1i0QIIQxdRE2jUCggk8mQnZ0NW1tbvewzr7AY3uG7AQDxkb1haaZzbq0W+D6IiKi6Ku/3t87XOSIiIiKqzRiOiIiIiDQwHBERERFpYDgiIiIi0sBwRERERKSB4YiIiIhIA8MRERERkQaGIyIiIiINDEdEREREGhiOiIiIiDQwHBERERFpYDgiIiIi0sBwRERERKSB4YiIiIhIg4mhCyCqjizNTJD4RT9Dl0FERAbAniMiIiIiDQxHRERERBoYjoiIiIg0MBwRERERaWA4IiIiItLAcERERESkgeGIiIiISAPDEREREZEGhiMiIiIiDQxHRERERBoYjoiIiIg0MBwRERERaWA4IiIiItLAcERERESkgeGIiIiISAPDEREREZEGhiMiIiIiDQxHRERERBoYjoiIiIg0MBwRERERaWA4IiIiItLAcERERESkgeGIiIiISEONCUezZs1CUFAQLC0tYWdnV+Y6Eomk1GPx4sVa65w9exZdu3aFhYUFGjZsiMjISAghquAdEBERUU1gYugCyquwsBCDBw9GYGAgli5d+sj1li9fjj59+qhfy2Qy9XOFQoGePXuiW7duiIqKwqVLlzB69GhYWVlh0qRJlVo/ERER1Qw1JhxFREQAAFasWPHY9ezs7ODk5FTmstWrVyM/Px8rVqyAVCqFj48PLl26hPnz5yM0NBQSiaTM7QoKClBQUKB+rVAoKvYmiIiIqNqrMafVyiskJARyuRzt27fH4sWLoVKp1MuOHTuGrl27QiqVqtt69+6N1NRUJCYmPnKfc+bMgUwmUz/c3Nwq8y0QERGRAdWqcPTZZ59h/fr12LdvH4YOHYpJkyZh9uzZ6uXp6elwdHTU2ubB6/T09EfuNywsDNnZ2epHSkpK5bwBIiIiMjiDnlabOXOm+nTZo0RFRcHf379c+5s+fbr6eZs2bQAAkZGRWu0Pnzp7MBj7UafUAEAqlWr1NhEREVHtZdBwFBISgqFDhz52HQ8PjwrvPyAgAAqFAjdv3oSjoyOcnJxK9RBlZGQAQKkeJSIiIqqbDBqO5HI55HJ5pe0/JiYG5ubm6qn/gYGB+OSTT1BYWAgzMzMAwJ49e+Di4vJUIYyIiIhqjxozWy05ORl37txBcnIylEolYmNjAQBNmjSBtbU1tm3bhvT0dAQGBsLCwgIHDhzAtGnT8NZbb6lPiQ0fPhwREREYPXo0PvnkE1y+fBmzZ89GeHj4Y0+rERERUd1RY8JReHg4Vq5cqX7t5+cHADhw4ACCg4NhamqKRYsWITQ0FCqVCo0bN0ZkZCTGjx+v3kYmk2Hv3r0YP348/P39Ua9ePYSGhiI0NLTK3w8RERFVTzUmHK1YseKx1zjq06eP1sUfH8XX1xeHDx/WY2VERERUm9SqqfxERERET4vhiIiIiEgDwxERERGRBoYjIiIiIg0MR0REREQaGI6IiIiINDAcEREREWlgOCIiIiLSwHBEREREpIHhiIiIiEgDwxERERGRBoYjIiIiIg0MR0REREQaTAxdANUulmYmSPyin6HLICIiqjD2HBERERFpYDgiIiIi0sBwRERERKSB4YiIiIhIA8MRERERkQaGIyIiIiINDEdEREREGhiOiIiIiDQwHBERERFpYDgiIiIi0sBwRERERKSB4YiIiIhIA8MRERERkQaGIyIiIiINDEdEREREGhiOiIiIiDQwHBERERFpYDgiIiIi0sBwRERERKSB4YiIiIhIA8MRERERkQaGIyIiIiINDEdEREREGmpMOJo1axaCgoJgaWkJOzu7UstXrFgBiURS5iMjIwMAkJiYWObyXbt2VfG7ISIiourKxNAFlFdhYSEGDx6MwMBALF26tNTyIUOGoE+fPlpto0ePRn5+Pho0aKDVvm/fPrRs2VL92t7evnKKJiIiohqnxoSjiIgIACU9RGWxsLCAhYWF+vWtW7ewf//+MoOUg4MDnJycKqVOIiIiqtlqzGk1Xf3yyy+wtLTEoEGDSi0bMGAAGjRogE6dOmHDhg1P3FdBQQEUCoXWg4iIiGqnWhuOli1bhuHDh2v1JllbW2P+/PnYsGEDdu7ciR49emDIkCFYtWrVY/c1Z84cyGQy9cPNza2yyyciIiIDMWg4mjlz5iMHUT94nDx5Uuf9Hjt2DPHx8RgzZoxWu1wux4cffogOHTrA398fkZGReO+99zB37tzH7i8sLAzZ2dnqR0pKis41ERERUc1g0DFHISEhGDp06GPX8fDw0Hm/S5YsQZs2bdCuXbsnrhsQEIAlS5Y8dh2pVAqpVKpzHURERFTzGDQcyeVyyOVyve4zNzcXv//+O+bMmVOu9WNiYuDs7KzXGoiIiKjmqjGz1ZKTk3Hnzh0kJydDqVQiNjYWANCkSRNYW1ur11u3bh2Ki4sxYsSIUvtYuXIlTE1N4efnByMjI2zbtg3fffcdvvzyy6p6G0RERFTN1ZhwFB4ejpUrV6pf+/n5AQAOHDiA4OBgdfvSpUsxcOBA1KtXr8z9fP7550hKSoKxsTGaNWuGZcuW4bXXXqvU2omIiKjmkAghhKGLqGkUCgVkMhmys7Nha2url33mFRbDO3w3ACA+sjcszWpMbiUiIqoRyvv9XWun8hMRERFVBLsnqglLMxMkftHP0GUQERHVeew5IiIiItLAcERERESkgeGIiIiISAPDEREREZEGhiMiIiIiDQxHRERERBoYjoiIiIg0MBwRERERaWA4IiIiItLAcERERESkgeGIiIiISAPDEREREZEGhiMiIiIiDQxHRERERBoYjoiIiIg0mBi6gJpICAEAUCgUBq6EiIiIyuvB9/aD7/FHYTiqgJycHACAm5ubgSshIiIiXeXk5EAmkz1yuUQ8KT5RKSqVCqmpqbCxsYFEItHbfhUKBdzc3JCSkgJbW1u97bc24zHTDY+Xbni8dMdjphseL909zTETQiAnJwcuLi4wMnr0yCL2HFWAkZERXF1dK23/tra2/EeiIx4z3fB46YbHS3c8Zrrh8dJdRY/Z43qMHuCAbCIiIiINDEdEREREGhiOqhGpVIoZM2ZAKpUaupQag8dMNzxeuuHx0h2PmW54vHRXFceMA7KJiIiINLDniIiIiEgDwxERERGRBoYjIiIiIg0MR0REREQaGI6q2KJFi+Dp6Qlzc3O0a9cOR44ceez6hw4dQrt27WBubo7GjRtj8eLFVVRp9aDL8dq0aRN69uyJ+vXrw9bWFoGBgdi9e3cVVls96Po79sBff/0FExMTtGnTpnILrGZ0PV4FBQWYNm0a3N3dIZVK8cwzz2DZsmVVVG31oOsxW716NVq3bg1LS0s4OzvjjTfewO3bt6uoWsM6fPgw+vfvDxcXF0gkEmzZsuWJ29Tlz31dj1elfe4LqjJr164Vpqam4ueffxbx8fFi4sSJwsrKSiQlJZW5/rVr14SlpaWYOHGiiI+PFz///LMwNTUVGzZsqOLKDUPX4zVx4kTx5Zdfin/++UdcunRJhIWFCVNTU3Hq1KkqrtxwdD1mD2RlZYnGjRuLXr16idatW1dNsdVARY7XgAEDRMeOHcXevXtFQkKCOHHihPjrr7+qsGrD0vWYHTlyRBgZGYkFCxaIa9euiSNHjoiWLVuKl156qYorN4ydO3eKadOmiY0bNwoAYvPmzY9dv65/7ut6vCrrc5/hqAp16NBBvPPOO1ptXl5e4uOPPy5z/SlTpggvLy+ttrffflsEBARUWo3Via7Hqyze3t4iIiJC36VVWxU9ZkOGDBHTp08XM2bMqFPhSNfj9X//939CJpOJ27dvV0V51ZKux+yrr74SjRs31mr77rvvhKura6XVWF2V58u+rn/uayrP8SqLPj73eVqtihQWFiI6Ohq9evXSau/Vqxf+/vvvMrc5duxYqfV79+6NkydPoqioqNJqrQ4qcrweplKpkJOTA3t7+8oosdqp6DFbvnw5rl69ihkzZlR2idVKRY7X1q1b4e/vj7lz56Jhw4Zo1qwZJk+ejPv371dFyQZXkWMWFBSE69evY+fOnRBC4ObNm9iwYQP69etXFSXXOHX5c18f9PW5zxvPVpHMzEwolUo4OjpqtTs6OiI9Pb3MbdLT08tcv7i4GJmZmXB2dq60eg2tIsfrYfPmzcO9e/fw6quvVkaJ1U5Fjtnly5fx8ccf48iRIzAxqVsfBxU5XteuXcPRo0dhbm6OzZs3IzMzE++99x7u3LlTJ8YdVeSYBQUFYfXq1RgyZAjy8/NRXFyMAQMGYOHChVVRco1Tlz/39UFfn/vsOapiEolE67UQolTbk9Yvq7220vV4PfDbb79h5syZWLduHRo0aFBZ5VVL5T1mSqUSw4cPR0REBJo1a1ZV5VU7uvyOqVQqSCQSrF69Gh06dEDfvn0xf/58rFixos70HgG6HbP4+HhMmDAB4eHhiI6Oxq5du5CQkIB33nmnKkqtker6535F6fNzv279qWhAcrkcxsbGpf66ysjIKPVXwgNOTk5lrm9iYgIHB4dKq7U6qMjxemDdunUYM2YM1q9fj+eee64yy6xWdD1mOTk5OHnyJGJiYhASEgKg5MtfCAETExPs2bMH3bt3r5LaDaEiv2POzs5o2LAhZDKZuq1FixYQQuD69eto2rRppdZsaBU5ZnPmzEGnTp3w0UcfAQBatWoFKysrdO7cGZ9//jl7Qh5Slz/3n4a+P/fZc1RFzMzM0K5dO+zdu1erfe/evQgKCipzm8DAwFLr79mzB/7+/jA1Na20WquDihwvoOQvh9GjR2PNmjV1bkyDrsfM1tYWZ8+eRWxsrPrxzjvvoHnz5oiNjUXHjh2rqnSDqMjvWKdOnZCamorc3Fx126VLl2BkZARXV9dKrbc6qMgxy8vLg5GR9leNsbExgP96ROg/dflzv6Iq5XP/qYZzk04eTIFdunSpiI+PFx988IGwsrISiYmJQgghPv74YzFy5Ej1+g+mdH744YciPj5eLF26tE5N6dT1eK1Zs0aYmJiI//3vfyItLU39yMrKMtRbqHK6HrOH1bXZaroer5ycHOHq6ioGDRok4uLixKFDh0TTpk3F2LFjDfUWqpyux2z58uXCxMRELFq0SFy9elUcPXpU+Pv7iw4dOhjqLVSpnJwcERMTI2JiYgQAMX/+fBETE6O+9AE/97Xperwq63Of4aiK/e9//xPu7u7CzMxMtG3bVhw6dEi97PXXXxddu3bVWv/gwYPCz89PmJmZCQ8PD/HDDz9UccWGpcvx6tq1qwBQ6vH6669XfeEGpOvvmKa6Fo6E0P14nT9/Xjz33HPCwsJCuLq6itDQUJGXl1fFVRuWrsfsu+++E97e3sLCwkI4OzuLESNGiOvXr1dx1YZx4MCBx34u8XNfm67Hq7I+9yVCsF+TiIiI6AGOOSIiIiLSwHBEREREpIHhiIiIiEgDwxERERGRBoYjIiIiIg0MR0REREQaGI6IiIiINDAcERERUbVw+PBh9O/fHy4uLpBIJNiyZYtO28+cORMSiaTUw8rKSqf9MBwRUbWUmJgIiUSC2NhYQ5dCRFXk3r17aN26Nb7//vsKbT958mSkpaVpPby9vTF48GCd9sNwRFTH3bp1C6ampsjLy0NxcTGsrKyQnJxs6LLg5uaGtLQ0+Pj4GLqUShUcHIwPPvjA4Psgqg6ef/55fP755xg4cGCZywsLCzFlyhQ0bNgQVlZW6NixIw4ePKhebm1tDScnJ/Xj5s2biI+Px5gxY3Sqw+Rp3gQR1XzHjh1DmzZtYGlpiRMnTsDe3h6NGjUydFkwNjaGk5PTI5cLIaBUKmFiwo8xorrijTfeQGJiItauXQsXFxds3rwZffr0wdmzZ9G0adNS6y9ZsgTNmjVD586ddfo57DkiquP+/vtvdOrUCQBw9OhR9fMnWb58OVq0aAFzc3N4eXlh0aJF6mUPTolt2rQJ3bp1g6WlJVq3bo1jx44BALKzs2FhYYFdu3Zp7XPTpk2wsrJCbm5uqdNqBw8ehEQiwe7du+Hv7w+pVIojR46goKAAEyZMQIMGDWBubo5nn30WUVFR6n0+2O7PP/+Ev78/LC0tERQUhIsXL6rXmTlzJtq0aYNly5ahUaNGsLa2xrvvvgulUom5c+fCyckJDRo0wKxZs7Tqzc7OxltvvYUGDRrA1tYW3bt3x+nTp0vt99dff4WHhwdkMhmGDh2KnJwcAMDo0aNx6NAhLFiwQD02IjExsczjvWjRIjRt2hTm5uZwdHTEoEGDnriP+Ph49O3bF9bW1nB0dMTIkSORmZmp3mdwcDBCQkIQEhICOzs7ODg4YPr06dC85eajfi5RVbt69Sp+++03rF+/Hp07d8YzzzyDyZMn49lnn8Xy5ctLrV9QUIDVq1fr3GsEAHiq29YSUY2UlJQkZDKZkMlkwtTUVJibmwuZTCbMzMyEVCoVMplMvPvuu4/c/qeffhLOzs5i48aN4tq1a2Ljxo3C3t5erFixQgghREJCggAgvLy8xPbt28XFixfFoEGDhLu7uygqKhJCCPHKK6+I1157TWu/r7zyihg2bJjWPmJiYoQQ/92tu1WrVmLPnj3iypUrIjMzU0yYMEG4uLiInTt3iri4OPH666+LevXqidu3b2tt17FjR3Hw4EERFxcnOnfuLIKCgtQ/d8aMGcLa2loMGjRIxMXFia1btwozMzPRu3dv8f7774sLFy6IZcuWCQDi2LFjQgghVCqV6NSpk+jfv7+IiooSly5dEpMmTRIODg7qn/1gvwMHDhRnz54Vhw8fFk5OTuKTTz4RQgiRlZUlAgMDxbhx40RaWppIS0sTxcXFpY53VFSUMDY2FmvWrBGJiYni1KlTYsGCBY/dR2pqqpDL5SIsLEycP39enDp1SvTs2VN069ZNvd+uXbsKa2trMXHiRHHhwgWxatUqYWlpKX766acn/lyiygZAbN68Wf36999/FwCElZWV1sPExES8+uqrpbZfs2aNMDExEWlpabr/7KcpnIhqpqKiIpGQkCBOnz4tTE1NRWxsrLhy5YqwtrYWhw4dEgkJCeLWrVuP3N7NzU2sWbNGq+2zzz4TgYGBQoj/gs2SJUvUy+Pi4gQAcf78eSGEEJs2bRLW1tbi3r17QgghsrOzhbm5udixY4fWPh4OR1u2bFHvMzc3V5iamorVq1er2woLC4WLi4uYO3eu1nb79u1Tr7Njxw4BQNy/f18IURJiLC0thUKhUK/Tu3dv4eHhIZRKpbqtefPmYs6cOUIIIf78809ha2sr8vPztY7DM888I3788cdH7vejjz4SHTt2VL/u2rWrmDhx4iOOdImNGzcKW1tbrf1oKmsfn376qejVq5dWW0pKigAgLl68qN6uRYsWQqVSqdeZOnWqaNGiRbl+LlFlejgcrV27VhgbG4sLFy6Iy5cvaz3KCkDdu3cXL730UoV+Nk+rEdVBJiYm8PDwwIULF9C+fXu0bt0a6enpcHR0RJcuXeDh4QG5XF7mtrdu3UJKSgrGjBkDa2tr9ePzzz/H1atXtdZt1aqV+rmzszMAICMjAwDQr18/mJiYYOvWrQCAjRs3wsbGBr169Xps7f7+/urnV69eRVFRkdapQFNTU3To0AHnz58vdy0A4OHhARsbG/VrR0dHeHt7w8jISKvtwTbR0dHIzc2Fg4OD1nFISEjQOg4P79fZ2Vnr55ZHz5494e7ujsaNG2PkyJFYvXo18vLyHrtNdHQ0Dhw4oFWbl5cXAGjVFxAQAIlEon4dGBiIy5cvQ6lUVujnElUWPz8/KJVKZGRkoEmTJlqPh8cnJiQk4MCBAxU7pQYOyCaqk1q2bImkpCQUFRVBpVLB2toaxcXFKC4uhrW1Ndzd3REXF1fmtiqVCgDw888/o2PHjlrLjI2NtV6bmpqqnz/4An6wvZmZGQYNGoQ1a9Zg6NChWLNmDYYMGfLEAdaa1ysR/46N0fxyf9D+cNvjanl4+YN1ymp7sI1KpYKzs7PWTJkH7OzsHrtfzZ9bHjY2Njh16hQOHjyIPXv2IDw8HDNnzkRUVJTWz9KkUqnQv39/fPnll6WWPQiHlfFziZ5Gbm4urly5on6dkJCA2NhY2Nvbo1mzZhgxYgRGjRqFefPmwc/PD5mZmdi/fz98fX3Rt29f9XbLli2Ds7Mznn/++QrVwZ4jojpo586diI2NhZOTE1atWoXY2Fj4+Pjg22+/RWxsLHbu3PnIbR0dHdGwYUNcu3at1F9vnp6eOtUxYsQI7Nq1C3FxcThw4ABGjBih0/ZNmjSBmZkZjh49qm4rKirCyZMn0aJFC532pau2bdsiPT0dJiYmpY7Do3rdymJmZgalUvnE9UxMTPDcc89h7ty5OHPmDBITE7F///5H7qNt27aIi4uDh4dHqfo0A+bx48e1tjt+/DiaNm2qDrqP+7lE+nby5En4+fnBz88PABAaGgo/Pz+Eh4cDKJkIMmrUKEyaNAnNmzfHgAEDcOLECbi5uan3oVKpsGLFCowePbrUH2zlxZ4jojrI3d0d6enpuHnzJl588UUYGRkhPj4eAwcOhIuLyxO3nzlzJiZMmABbW1s8//zzKCgowMmTJ3H37l2EhoaWu46uXbvC0dERI0aMgIeHBwICAnR6H1ZWVnj33Xfx0UcfqS9BMHfuXOTl5VW4O728nnvuOQQGBuKll17Cl19+iebNmyM1NRU7d+7ESy+9pHX673E8PDxw4sQJJCYmwtraGvb29lqn8gBg+/btuHbtGrp06YJ69eph586dUKlUaN68+SP3MX78ePz8888YNmwYPvroI8jlcly5cgVr167Fzz//rP7SSElJQWhoKN5++22cOnUKCxcuxLx588r1c4n0LTg4WGu25MNMTU0RERGBiIiIR65jZGSElJSUp6qDPUdEddTBgwfRvn17mJub48SJE2jYsGG5ghEAjB07FkuWLMGKFSvg6+uLrl27YsWKFTr3HEkkEgwbNgynT5/WudfogS+++AKvvPIKRo4cibZt2+LKlSvYvXs36tWrV6H9lZdEIsHOnTvRpUsXvPnmm2jWrBmGDh2KxMREODo6lns/kydPhrGxMby9vVG/fv0yL8BpZ2eHTZs2oXv37mjRogUWL16M3377DS1btnzkPlxcXPDXX39BqVSid+/e8PHxwcSJEyGTybTC16hRo3D//n106NAB48ePx/vvv4+33nqrXD+XqLaSiMdFNCIiqrWCg4PRpk0bfPvtt4YuhahaYc8RERERkQaGIyIiIiINPK1GREREpIE9R0REREQaGI6IiIiINDAcEREREWlgOCIiIiLSwHBEREREpIHhiIiIiEgDwxERERGRBoYjIiIiIg3/D5omjKY8WL2OAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "train_fn = functools.partial(\n", " ppo.train, num_timesteps=10_000_000, num_evals=10, reward_scaling=0.1,\n", " episode_length=1000, normalize_observations=True, action_repeat=1,\n", " unroll_length=10, num_minibatches=32, num_updates_per_batch=8,\n", " discounting=0.97, learning_rate=3e-4, entropy_cost=1e-3, num_envs=1024,\n", " batch_size=1024, seed=0)\n", "\n", "x_data = []\n", "y_data = []\n", "ydataerr = []\n", "env = envs.get_environment(\"trotting_anymal\", step_k = 13)\n", "\n", "def progress(num_steps, metrics):\n", " x_data.append(num_steps)\n", " y_data.append(metrics['eval/episode_reward'])\n", " ydataerr.append(metrics['eval/episode_reward_std'])\n", "\n", "make_inference_fn, params, _= train_fn(environment=env, progress_fn=progress)\n", "\n", "plt.errorbar(x_data, y_data, yerr=ydataerr)\n", "plt.xlabel('# environment steps')\n", "plt.ylabel('reward per episode')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that PPO takes around 9e6 simulator steps to catch up to APG." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Study 2: Quadruped Locomotion\n", "\n", "As we saw in the imitation learning example, FoPG methods benefit from detailed reward signals. To teach locomotion, we reward the feet based on the Raibert Heuristic. We implement a gait schedule, that incentivizes opposite pairs of legs to move in sync at a fixed frequency. At the beginning of a new scheduled step, we calculate the target position for the feet at the end of the step. \n", "\n", "For each foot, we calculate:\n", "\n", "$$\n", "p^* = h_0 + \\frac{\\Delta T}{2} v_0\n", "$$\n", "\n", "Where $p^*$ is x, y component of the foot's target position, $h_0$ is the x, y component of the corresponding hip at lift-off, $\\Delta T$ is the scheduled step duration, and $v_0$ is the base velocity at lift-off. \n", "\n", "Due to their limited exploration power, FoPG methods benefit greatly from having an good \"initial guess\" of the policy - familiar terminology in Model Predictive Control and Trajectory Optimization. We formulate the problem as [residual learning](https://arxiv.org/abs/1512.03385). Let $\\phi$ be the parameters of a baseline policy that we already have, and let $f$ and $g$ be the neural networks for the learned and baseline policy. We freeze $\\phi$ and learn parameters $\\theta$, for the policy:\n", "\n", "$$\n", "a_t = f(g(x_t; \\phi), x_t; \\theta) + g(x_t; \\phi)\n", "$$\n", "\n", "We use the in-place trotting policy from last section as $\\phi$ and $x_t$ denotes the state at time t. In this example we track a 0.75 m/s velocity target, but since locomotion is stabler at faster trots, you can experiment with $\\phi$ for faster velocity targets!\n", "\n", "While it might seem more natural to \"hotstart\" the learning by simply initializing the parameters $\\theta$ as $\\phi$, with the policy $a_t = f(x_t; \\theta)$, we find that the residual method trains more stably in practice." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### The RL Environment" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "def axis_angle_to_quaternion(v: jp.ndarray, theta:jp.float_):\n", " \"\"\" \n", " axis angle representation: rotation of theta around v. \n", " \"\"\" \n", " return jp.concatenate([jp.cos(0.5*theta).reshape(1), jp.sin(0.5*theta)*v.reshape(3)])\n", "\n", "def get_config():\n", " \"\"\"Returns reward config for anymal quadruped environment.\"\"\"\n", "\n", " def get_default_rewards_config():\n", " default_config = config_dict.ConfigDict(\n", " dict(\n", " scales=config_dict.ConfigDict(\n", " dict(\n", " tracking_lin_vel = 1.0,\n", " orientation = -1.0, # non-flat base\n", " height = 0.5,\n", " lin_vel_z=-1.0, # prevents the suicide policy\n", " torque = -0.01,\n", " feet_pos = -1, # Bad action hard-coding. \n", " feet_height = -1, # prevents it from just standing still\n", " joint_velocity = -0.001\n", " )\n", " ),\n", " )\n", " )\n", " return default_config\n", "\n", " default_config = config_dict.ConfigDict(\n", " dict(rewards=get_default_rewards_config(),))\n", "\n", " return default_config\n", "\n", "class DiffAnymal(PipelineEnv):\n", "\n", " def __init__(\n", " self,\n", " termination_height: float=0.25,\n", " **kwargs,\n", " ):\n", " \n", " self.target_vel = kwargs.pop('target_vel', 0.75)\n", " step_k = kwargs.pop('step_k', 25)\n", " self.baseline_inference_fn = kwargs.pop(\"baseline_inference_fn\")\n", " physics_steps_per_control_step = 10\n", " kwargs['n_frames'] = kwargs.get(\n", " 'n_frames', physics_steps_per_control_step)\n", " self.termination_height = termination_height\n", "\n", " mj_model = mujoco.MjModel.from_xml_path(xml_path)\n", " self._init_q = mj_model.keyframe('standing').qpos\n", " self._default_ap_pose = mj_model.keyframe('standing').qpos[7:]\n", " self.reward_config = get_config()\n", "\n", " self.action_loc = self._default_ap_pose\n", " self.action_scale = jp.array([0.2, 0.8, 0.8] * 4)\n", " \n", " self.target_h = self._init_q[2]\n", "\n", " sys = mjcf.load_model(mj_model)\n", " super().__init__(sys=sys, **kwargs)\n", " \n", " \"\"\"\n", " Kinematic references are used for gait scheduling.\n", " \"\"\"\n", "\n", " kinematic_ref_qpos = make_kinematic_ref(\n", " cos_wave, step_k, scale=0.3, dt=self.dt)\n", " self.l_cycle = jp.array(kinematic_ref_qpos.shape[0])\n", "\n", " # Expand to base state\n", " kinematic_ref_qpos += self._default_ap_pose\n", " ref_qs = np.tile(self._init_q.reshape(1, 19), (self.l_cycle, 1))\n", " ref_qs[:, 7:] = kinematic_ref_qpos\n", " self.kinematic_ref_qpos = jp.array(ref_qs)\n", "\n", " \"\"\"\n", " Foot tracking\n", " \"\"\"\n", " gait_k = step_k * 2\n", " self.gait_period = gait_k * self.dt\n", "\n", " self.step_k = step_k\n", " self.feet_inds = jp.array([21,28,35,42]) # LF, RF, LH, RH\n", " self.hip_inds = self.feet_inds - 6\n", "\n", " self.pipeline_step = jax.checkpoint(self.pipeline_step,\n", " policy=jax.checkpoint_policies.dots_with_no_batch_dims_saveable)\n", " \n", " def reset(self, rng: jax.Array) -> State:\n", " rng, key_xyz, key_ang, key_ax, key_q, key_qd = jax.random.split(rng, 6)\n", "\n", " qpos = jp.array(self._init_q)\n", " qvel = jp.zeros(18)\n", " \n", " #### Add Randomness ####\n", " \n", " r_xyz = 0.2 * (jax.random.uniform(key_xyz, (3,))-0.5)\n", " r_angle = (jp.pi/12) * (jax.random.uniform(key_ang, (1,)) - 0.5) # 15 deg range\n", " r_axis = (jax.random.uniform(key_ax, (3,)) - 0.5)\n", " r_axis = r_axis / jp.linalg.norm(r_axis)\n", " r_quat = axis_angle_to_quaternion(r_axis, r_angle)\n", "\n", " r_joint_q = 0.2 * (jax.random.uniform(key_q, (12,)) - 0.5)\n", " r_joint_qd = 0.1 * (jax.random.uniform(key_qd, (12,)) - 0.5)\n", " \n", " qpos = qpos.at[0:3].set(qpos[0:3] + r_xyz)\n", " qpos = qpos.at[3:7].set(r_quat)\n", " qpos = qpos.at[7:19].set(qpos[7:19] + r_joint_q)\n", " qvel = qvel.at[6:18].set(qvel[6:18] + r_joint_qd)\n", " \n", " data = self.pipeline_init(qpos, qvel)\n", "\n", " # Ensure you're not sunken into the ground nor above it.\n", " pen = jp.min(data.contact.dist)\n", " qpos = qpos.at[2].set(qpos[2] - pen)\n", " data = self.pipeline_init(qpos, qvel)\n", "\n", " state_info = {\n", " 'rng': rng,\n", " 'steps': 0.0,\n", " 'reward_tuple': {\n", " 'tracking_lin_vel': 0.0,\n", " 'orientation': 0.0,\n", " 'height': 0.0,\n", " 'lin_vel_z': 0.0,\n", " 'torque': 0.0,\n", " 'joint_velocity': 0.0,\n", " 'feet_pos': 0.0,\n", " 'feet_height': 0.0\n", " },\n", " 'last_action': jp.zeros(12), # from MJX tutorial.\n", " 'baseline_action': jp.zeros(12),\n", " 'xy0': jp.zeros((4, 2)),\n", " 'k0': 0.0,\n", " 'xy*': jp.zeros((4, 2))\n", " }\n", "\n", " x, xd = data.x, data.xd\n", " _obs = self._get_obs(data.qpos, x, xd, state_info) # inner obs; to trotter\n", " \n", " action_key, key = jax.random.split(state_info['rng'])\n", " state_info['rng'] = key\n", " next_action, _ = self.baseline_inference_fn(_obs, action_key)\n", "\n", " obs = jp.concatenate([_obs, next_action])\n", "\n", " reward, done = jp.zeros(2)\n", " metrics = {}\n", " for k in state_info['reward_tuple']:\n", " metrics[k] = state_info['reward_tuple'][k]\n", " state = State(data, obs, reward, done, metrics, state_info)\n", " return jax.lax.stop_gradient(state)\n", "\n", " def step(self, state: State, action: jax.Array) -> State:\n", "\n", " action = jp.clip(action, -1, 1)\n", "\n", " cur_base = state.obs[-12:]\n", " action += cur_base\n", " state.info['baseline_action'] = cur_base\n", "\n", " action = self.action_loc + (action * self.action_scale)\n", "\n", " data = self.pipeline_step(state.pipeline_state, action)\n", " \n", " # observation data\n", " x, xd = data.x, data.xd\n", " obs = self._get_obs(data.qpos, x, xd, state.info)\n", "\n", " # Terminate if flipped over or fallen down.\n", " done = 0.0\n", " done = jp.where(x.pos[0, 2] < self.termination_height, 1.0, done)\n", " up = jp.array([0.0, 0.0, 1.0])\n", " done = jp.where(jp.dot(math.rotate(up, x.rot[0]), up) < 0, 1.0, done)\n", "\n", " #### Foot Position Reference Updating ####\n", "\n", " # Detect the start of a new step\n", " s = state.info['steps']\n", " step_num = s // (self.step_k)\n", " even_step = step_num % 2 == 0\n", " new_step = (s % self.step_k) == 0\n", " new_even_step = jp.logical_and(new_step, even_step)\n", " new_odd_step = jp.logical_and(new_step, jp.logical_not(even_step))\n", "\n", " # Apply railbert heuristic to calculate target foot position, after step\n", " hip_xy = data.geom_xpos[self.hip_inds][:,:2] # 4 x 2\n", " v_body = data.qvel[0:2]\n", " step_period = self.gait_period/2\n", " raibert_xy = hip_xy + (step_period/2) * v_body\n", "\n", " # Update. \n", " cur_tars = state.info['xy*']\n", " i_RFLH = jp.array([1, 2])\n", " i_LFRH = jp.array([0, 3])\n", " feet_xy = data.geom_xpos[self.feet_inds][:,:2]\n", " \n", " # With the trotting gait, we will move one pair of opposite legs, \n", " # and keep the other pair fixed in place.\n", " case_c1 = raibert_xy.at[i_LFRH].set(feet_xy[i_LFRH]) \n", " case_c2 = raibert_xy.at[i_RFLH].set(feet_xy[i_RFLH])\n", " xy_tars = jp.where(new_even_step, case_c1, cur_tars)\n", " xy_tars = jp.where(new_odd_step, case_c2, xy_tars)\n", " state.info['xy*'] = xy_tars\n", "\n", " # Save timestep and location at start of step.\n", " state.info['k0'] = jp.where(new_step,\n", " state.info['steps'],\n", " state.info['k0'])\n", " state.info['xy0'] = jp.where(new_step, \n", " feet_xy,\n", " state.info['xy0'])\n", "\n", " # reward\n", " reward_tuple = {\n", " 'tracking_lin_vel': (\n", " self._reward_tracking_lin_vel(jp.array([self.target_vel, 0, 0]), x, xd)\n", " * self.reward_config.rewards.scales.tracking_lin_vel\n", " ),\n", " 'orientation': (\n", " self._reward_orientation(x)\n", " * self.reward_config.rewards.scales.orientation\n", " ),\n", " 'lin_vel_z': (\n", " self._reward_lin_vel_z(xd)\n", " * self.reward_config.rewards.scales.lin_vel_z\n", " ),\n", " 'height': (\n", " self._reward_height(data.qpos) \n", " * self.reward_config.rewards.scales.height\n", " ),\n", " 'torque': (\n", " self._reward_action(data.qfrc_actuator)\n", " * self.reward_config.rewards.scales.torque\n", " ),\n", " 'joint_velocity': (\n", " self._reward_joint_velocity(data.qvel)\n", " * self.reward_config.rewards.scales.joint_velocity\n", " ),\n", " 'feet_pos': (\n", " self._reward_feet_pos(data, state)\n", " * self.reward_config.rewards.scales.feet_pos\n", " ),\n", " 'feet_height': (\n", " self._reward_feet_height(data, state.info)\n", " * self.reward_config.rewards.scales.feet_height\n", " )\n", " }\n", " \n", " reward = sum(reward_tuple.values())\n", "\n", " # state management\n", " state.info['reward_tuple'] = reward_tuple\n", " state.info['last_action'] = action\n", "\n", " for k in state.info['reward_tuple'].keys():\n", " state.metrics[k] = state.info['reward_tuple'][k]\n", "\n", " # next action\n", " action_key, key = jax.random.split(state.info['rng'])\n", " state.info['rng'] = key\n", " next_action, _ = self.baseline_inference_fn(obs, action_key)\n", " obs = jp.concatenate([obs, next_action])\n", "\n", " state = state.replace(\n", " pipeline_state=data, obs=obs, reward=reward,\n", " done=done)\n", " return state\n", "\n", " def _get_obs(self, qpos: jax.Array, x: Transform, xd: Motion,\n", " state_info: Dict[str, Any]) -> jax.Array:\n", "\n", " inv_base_orientation = math.quat_inv(x.rot[0])\n", " local_rpyrate = math.rotate(xd.ang[0], inv_base_orientation)\n", "\n", " obs_list = []\n", " # yaw rate\n", " obs_list.append(jp.array([local_rpyrate[2]]) * 0.25)\n", " # projected gravity\n", " obs_list.append(\n", " math.rotate(jp.array([0.0, 0.0, -1.0]), inv_base_orientation))\n", " # motor angles\n", " angles = qpos[7:19]\n", " obs_list.append(angles - self._default_ap_pose)\n", " # last action\n", " obs_list.append(state_info['last_action'])\n", " # base action\n", " cur_base = self.kinematic_ref_qpos[jp.array(state_info['steps']%self.l_cycle, int)]\n", " obs_list.append(cur_base)\n", "\n", " obs = jp.clip(jp.concatenate(obs_list), -100.0, 100.0)\n", "\n", " return obs\n", "\n", " # ------------ reward functions----------------\n", " def _reward_tracking_lin_vel(\n", " self, commands: jax.Array, x: Transform, xd: Motion) -> jax.Array:\n", " # Tracking of linear velocity commands (xy axes)\n", " local_vel = math.rotate(xd.vel[0], math.quat_inv(x.rot[0]))\n", " lin_vel_error = jp.sum(jp.square(commands[:2] - local_vel[:2]))\n", " lin_vel_reward = jp.exp(-lin_vel_error)\n", " return lin_vel_reward\n", " def _reward_orientation(self, x: Transform) -> jax.Array:\n", " # Penalize non flat base orientation\n", " up = jp.array([0.0, 0.0, 1.0])\n", " rot_up = math.rotate(up, x.rot[0])\n", " return jp.sum(jp.square(rot_up[:2]))\n", " def _reward_lin_vel_z(self, xd: Motion) -> jax.Array:\n", " # Penalize z axis base linear velocity\n", " return jp.clip(jp.square(xd.vel[0, 2]), 0, 10)\n", " def _reward_joint_velocity(self, qvel):\n", " return jp.clip(jp.sqrt(jp.sum(jp.square(qvel[6:]))), 0, 100)\n", " def _reward_height(self, qpos) -> jax.Array:\n", " return jp.exp(-jp.abs(qpos[2] - self.target_h)) # Not going to be > 1 meter tall.\n", " def _reward_action(self, action) -> jax.Array:\n", " return jp.sqrt(jp.sum(jp.square(action)))\n", " def _reward_feet_pos(self, data, state): \n", " dt = (state.info['steps'] - state.info['k0']) * self.dt # scalar\n", " step_period = self.gait_period / 2\n", " xyt = state.info['xy0'] + (state.info['xy*'] - state.info['xy0']) * (dt/step_period)\n", "\n", " feet_pos = data.geom_xpos[self.feet_inds][:, :2]\n", "\n", " rews = jp.sum(jp.square(feet_pos - xyt), axis=1) \n", " rews = jp.clip(rews, 0, 10)\n", " return jp.sum(rews)\n", " def _reward_feet_height(self, data, state_info):\n", " \"\"\" \n", " Feet height tracks rectified sine waves \n", " \"\"\"\n", " h_tar = 0.1\n", " t = state_info['steps'] * self.dt\n", " offset = self.gait_period/2\n", " ref1 = jp.sin((2*jp.pi/self.gait_period)*t) # RF and LH feet\n", " ref2 = jp.sin((2*jp.pi/self.gait_period)*(t - offset)) # LF and RH\n", " \n", " ref1, ref2 = ref1 * h_tar, ref2 * h_tar\n", " h_tars = jp.array([ref2, ref1, ref1, ref2])\n", " h_tars = h_tars.clip(min=0, max=None) + 0.02 # offset height of feet.\n", " \n", " feet_height = data.geom_xpos[self.feet_inds][:,2]\n", " errs = jp.clip(jp.square(feet_height - h_tars), 0, 10)\n", " return jp.sum(errs)\n", " \n", "envs.register_environment('anymal', DiffAnymal)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Residual Learning via FoPG's\n", "Takes 15 minutes on a NVIDIA 3060 TI GPU" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# Reconstruct the trotting inference function\n", "make_networks_factory = functools.partial(\n", " apg_networks.make_apg_networks,\n", " hidden_layer_sizes=(256, 128)\n", ")\n", "\n", "nets = make_networks_factory(observation_size=47,\n", " action_size=12,\n", " preprocess_observations_fn=running_statistics.normalize)\n", "\n", "make_inference_fn = apg_networks.make_inference_fn(nets)\n", "\n", "# Configure locomotion training\n", "make_networks_factory = functools.partial(\n", " apg_networks.make_apg_networks,\n", " hidden_layer_sizes=(128, 64)\n", ")\n", "\n", "epochs = 499\n", "\n", "train_fn = functools.partial(apg.train,\n", " episode_length=1000,\n", " policy_updates=epochs,\n", " horizon_length=32,\n", " num_envs=64,\n", " learning_rate=1.5e-4,\n", " schedule_decay=0.995,\n", " num_eval_envs=64,\n", " num_evals=10 + 1,\n", " use_float64=True,\n", " normalize_observations=True,\n", " network_factory=make_networks_factory)\n", "\n", "model_path = '/tmp/trotting_2hz_policy'\n", "params = model.load_params(model_path)\n", "baseline_inference_fn = make_inference_fn(params)\n", "\n", "env_kwargs = dict(target_vel=0.75, step_k=13, \n", " baseline_inference_fn=baseline_inference_fn)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjMAAAGdCAYAAADnrPLBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA970lEQVR4nO3deXzU1b3/8XfWyUIyZCEZAgECpGxBwWARREFZtLJcr7dSRSn+il69KBrBjdqqpdegWNEWWhTrT6yI8OuCequ1BPWiFFSIBNkRCBCWGJYwk3Ummfn+/kgyMuxLJt/MzOv5eMzDme+cmXxmhHzfnHO+54QZhmEIAAAgQIWbXQAAAMClIMwAAICARpgBAAABjTADAAACGmEGAAAENMIMAAAIaIQZAAAQ0AgzAAAgoEWaXUBL8Hg8OnjwoBISEhQWFmZ2OQAA4DwYhqGKigplZGQoPPzM/S8hEWYOHjyozMxMs8sAAAAXoaSkRB07djzj8yERZhISEiQ1fBmJiYkmVwMAAM6Hw+FQZmam9zx+JiERZpqGlhITEwkzAAAEmHNNEWECMAAACGiEGQAAENAIMwAAIKARZgAAQEAjzAAAgIB20WHms88+09ixY5WRkaGwsDC9++67Ps8bhqFnnnlGGRkZio2N1bBhw7R582afNk6nU1OnTlVqaqri4+M1btw47d+/36dNeXm5Jk6cKKvVKqvVqokTJ+r48eMXWzYAAAgyFx1mqqqqdPnll2vevHmnfX727NmaM2eO5s2bp7Vr18pms2nkyJGqqKjwtsnLy9OyZcu0ZMkSrVq1SpWVlRozZozcbre3zYQJE1RUVKSPPvpIH330kYqKijRx4sSLLRsAAAQboxlIMpYtW+Z97PF4DJvNZjz33HPeY7W1tYbVajVeeeUVwzAM4/jx40ZUVJSxZMkSb5sDBw4Y4eHhxkcffWQYhmFs2bLFkGR88cUX3jZr1qwxJBnbtm077/rsdrshybDb7Rf7EQEAQAs73/O3X+bMFBcXq7S0VKNGjfIes1gsGjp0qFavXi1JKiwsVF1dnU+bjIwM5eTkeNusWbNGVqtVAwcO9La56qqrZLVavW1Ox+l0yuFw+NwAAEBw8kuYKS0tlSSlp6f7HE9PT/c+V1paqujoaCUlJZ21TVpa2invn5aW5m1zOrNmzfLOsbFarezLBABAEPPr1UwnLz9sGMY5lyQ+uc3p2p/rfWbMmCG73e69lZSUXGDlAAAgUPglzNhsNkk6pfekrKzM21tjs9nkcrlUXl5+1jbffffdKe9/+PDhU3p9TmSxWLz7MLEfEwAAwc0vYSYrK0s2m00FBQXeYy6XSytXrtTgwYMlSbm5uYqKivJpc+jQIW3atMnbZtCgQbLb7frqq6+8bb788kvZ7XZvGwAAENouetfsyspK7dy50/u4uLhYRUVFSk5OVqdOnZSXl6f8/HxlZ2crOztb+fn5iouL04QJEyRJVqtVkydP1vTp05WSkqLk5GQ98sgj6tu3r0aMGCFJ6tWrl2688Ubdc889evXVVyVJ//mf/6kxY8aoR48el/K5AQDAJap21av3U/+UJG2ZeYPioi86VlySi/6p69at03XXXed9PG3aNEnSpEmTtHDhQj322GOqqanRlClTVF5eroEDB2r58uVKSEjwvuall15SZGSkxo8fr5qaGg0fPlwLFy5URESEt83bb7+tBx980HvV07hx4864tg0AAPAvwzBU6azX4QqnSo5Ve49vOejQgC7JptQUZhiGYcpPbkEOh0NWq1V2u535MwAAnIaz3q0jlS4drnD63iprT7jf8N/aOs8pr58+8geaOjy7WWs63/O3Of1BAADA7zweQ+XVLm8I8Q0p398vq3DKXlN3Qe+dYIlUSpto7Tna0DvTOSXOHx/hvBBmAABoBS5k/klV4zDP6UJKWUWt9/iRSpfcnvMfgImKCFO7Nha1Szjh5vM4RmkJFqW2sSg2OsKn5hG9z3yVsb8RZgAAaAVOnPWxcvthOWrrVOY4KbA03q92uc/yTqdKjo8+JaSknSasWGOjzrke3InioiO157nRF1SLPxBmAABoIfVujw4er9W+Y9Xae6xK+45Va9/Rau09Wq19J0ym/a+3vz7ne8VGRSgt8fShpOFxjNolWJTSJlpREX5dI9d0hBkAAJpRpbNe+45Wa19jWGkKKvuOVetAeY3qz2PYp09GotITY04NKCeElngLp/AmfBMAAFwAwzBUVuH8PqQcrdLeY9XeXpajVa6zvj46MlyZSbHqnBKvTslx6pQcp84pcUpLtGjs3H9Jkv583yDT1mwJRHxTAAJWa1mwC8HHWe9WybEalRyr1t7GsFLS2MtSUl592kuTT5QUF6VOKfHq3BhWOqXENdxPiVN6QozCw08/L6U1zD8JRPzNBwCEHMMwdLy6rnHuSkPvyolDQqWOWp1tFbaI8DBltI1R5+R4ZTb2rHROjlNmY2BJjIlquQ8DwgwAIPic2Gv3+qQBJwwLfR9aKmrrz/oe8dER6pQSr07Jpw4JZbSNDfpJtYGEMAMACEhVznodOF7TcCuv0cHG+weP16ikvMbbbvKb6874HumJlsaQEu8NKp1SGkJLSnz0BV2mDPMQZgAArY7HY+hIlbMxpNTqwPFqHTxeq/0nhJbzXbE2KzVeXVKa5q7Ee+euZCbFKTY64txvgFaPMAMAaHHOercOHa89Y8/KQXutXPVnn2QrSYkxkeqQFKcObWPUoW2sMtrGqkNSrFLio3X7a19Kkj54cAiTw4Mc/3cBAGd1oVeNGYYhe03daUJKrfY3hpXDFc5z/tzwMCk9MaYhoJwQVDo23s9oG6OEs0y05cqg0EGYAQBckHq3R99VOBtCSnmNt3fl4Anhpeo8ltuPiQr3hpSOSbHKsDaElabwYrPGMMkW54UwAwDwUVvn9vamHCivUfGRKu9zw19cqbIK53ltXpgSH90QTk4KKR0ae1iS4i5sHyDgTAgzABBivFcBlddo//Ea7S9vWGZ/f2Mvy9mGgA7ZayVJkeFhan/CPJWOJwwDNYWWmCgm16JlEGYAIMjYa+oaw0n196GlMajsL69WefW5rwKKjYpQx6SGcJKeGKOla0skSYvvHqiu7dqoXYJFEWdYxRZoaYQZAAgghmHoWJXrtCGl6f65FoOTpISYSHVMilOHxvkqHZNiG+/HnTIEVO2q94aZfp3acmUQWh3+RAJACzrXlUEej6EjlU6VnBBSDnjvNwSYmrpzT65Njo/2BpSmwNKhMbx0SIqVNZbl9hE8CDMAYJL3iw7qcIXTp3fl4PFaudznXl8lLcHScJnyCb0rTZctd0iKbdbek7joSC5zRqtGmAEAP7PX1Omr4mNas+uo/rXzsPf4E3/beNr24WFSe2usb0hJilWHtg1DQO2tMUyuBU5AmAGAZlbprNfaPcf0xa6jWrP7qDYdsOt0VzIPzEpW55Q4b0hpGhZifRXgwhBmAOAS1da5Vbi3XKt3HdGaXUf1zX676k9KL11T43VVtxTldmqr6X/+RpL0xv+5ksm0QDPgbxEAXCBnvVtF+45rze6jWr3rqIr2HT9lnktmcqwGdU3RoG4pGtQ1VTZrjKSGCcBNYQZA8yDMAMA51Lk9+ma/XV/sPqo1u45q3d5jqq3zDS+2xBgN7paiq7qlaFDXFGUmx532vZhMCzQ/wgwAnMTtMbTloKNh2Gj3Ua0tPnbKXkOpbaJ1VdcUDe6WqkHdUtQlJY6l+QGTEGYAhDyPx9D27yq0ZlfDsNFXxUflOGnhubZxUboqq2HYaHC3FHVPa0N4AVoJwgyAkGMYhnYdrtSaxquNvth9TMeqXD5tEiyRGtg1WVc1znvpZUtUOMv3A60SYQZA0DMMQ/uOVWv1rqPeAHPyZopx0RG6skty44TdFPXJSFQkl0cDAYEwAyAoHThe0zhsdERf7Dqqg427PTexRIYrt3OSBndr6Hm5rGNb1nYBAhRhBkBQOFzhVFHJd955L/uOVfs8HxURpv6ZSbqqcc5Lv8y2rKILBAnCDICAZa+p894f+sL/+jwXER6myzpaNajxiqPczkmKjSa8AMGIMAMg4NS7PXrnq316sWCH91hYmJSTYfXOebkyK1ltLPyKA0IBf9MBBJR/7Tyimf+zRdu/q/A5vvqJ69XeGmtSVQDMxGw3AAFhz5Eq3fOndbrjj19q+3cVahsXpV+O6eV93hobZWJ1AMxEzwyAVq2itk7zPt2pN1btkcvtUUR4mCZe1Vl5I7IVHRmuX/99q9klAjAZYQZAq+T2GPpLYYle+OcOHalsWBPm2h+001Njeql7WoKkhk0bAYAwA6DV+ar4mGb+fbM2HXBIkrqmxusXY3rpuh5pbCEA4BSEGQCtxv7yas36xzZ98M0hSVJCTKQeGp6tnw7qouhIpvgBOD3CDADTVbvq9cr/7tKrn+2Ws96j8DDpth920vSRP1BKG4vZ5QFo5QgzAEzj8Rh6b8MBPf+P7Sp1NGw3MKhrip4a21u92ieaXB2AQOHXftv6+nr94he/UFZWlmJjY9W1a1fNnDlTHo/H28YwDD3zzDPKyMhQbGyshg0bps2bN/u8j9Pp1NSpU5Wamqr4+HiNGzdO+/fv92fpAPysqOS4/uOV1Xp46QaVOmqVmRyrV+68QovvGUiQAXBB/Bpmnn/+eb3yyiuaN2+etm7dqtmzZ+uFF17Q3LlzvW1mz56tOXPmaN68eVq7dq1sNptGjhypiorvF8TKy8vTsmXLtGTJEq1atUqVlZUaM2aM3G63P8sH4Ael9lpNW1qkm3//L63fd1zx0RF67MYeKnh4qG7Mac8EXwAXzK/DTGvWrNG//du/afTo0ZKkLl266J133tG6deskNfTKvPzyy3ryySd1yy23SJLefPNNpaena/Hixbr33ntlt9v1+uuv66233tKIESMkSYsWLVJmZqZWrFihG264wZ8fAUAzqa1z64+f79bvP92lmrqGf4j8OLejHruhh9ISY0yuDkAg82vPzJAhQ/Txxx9rx46G/VM2bNigVatW6aabbpIkFRcXq7S0VKNGjfK+xmKxaOjQoVq9erUkqbCwUHV1dT5tMjIylJOT421zMqfTKYfD4XMDYA7DMPTBN4c0/MWV+s3yHaqpcyu3c5Lef+Bq/ebWywkyAC6ZX3tmHn/8cdntdvXs2VMRERFyu9169tlndfvtt0uSSktLJUnp6ek+r0tPT9fevXu9baKjo5WUlHRKm6bXn2zWrFn61a9+1dwfB8AF2nTArpl/36Kvio9JktpbYzTjpl4aexnDSQCaj1/DzNKlS7Vo0SItXrxYffr0UVFRkfLy8pSRkaFJkyZ52538S80wjHP+ojtbmxkzZmjatGnexw6HQ5mZmZfwSQBciMMVTr24fLuWriuRYUgxUeG6b2g33XttN8VGR5hdHoAg49cw8+ijj+qJJ57QbbfdJknq27ev9u7dq1mzZmnSpEmy2WySGnpf2rdv731dWVmZt7fGZrPJ5XKpvLzcp3emrKxMgwcPPu3PtVgsslhYmwJoaa56jxauLtbvPt6pSmfDVgPjLs/QEz/qqYy27GgNwD/8Omemurpa4eG+PyIiIsJ7aXZWVpZsNpsKCgq8z7tcLq1cudIbVHJzcxUVFeXT5tChQ9q0adMZwwyAlmUYhgq2fKdRL61U/ofbVOms12Udrfrrfw3S727vT5AB4Fd+7ZkZO3asnn32WXXq1El9+vTR+vXrNWfOHP3sZz+T1DC8lJeXp/z8fGVnZys7O1v5+fmKi4vThAkTJElWq1WTJ0/W9OnTlZKSouTkZD3yyCPq27ev9+omAObZXlqhX/99i1btPCJJapdg0eM39tQt/TsoPJx5MQD8z69hZu7cufrlL3+pKVOmqKysTBkZGbr33nv11FNPeds89thjqqmp0ZQpU1ReXq6BAwdq+fLlSkhI8LZ56aWXFBkZqfHjx6umpkbDhw/XwoULFRHB2DvQXKpd9er91D8lSVtm3qC46LP/eiivcumlFTv09pf75PYYio4M191DsjTluu5qY2FxcQAtJ8wwDMPsIvzN4XDIarXKbrcrMZGVRYHTOd8wU+f2aNEXe/Xyim9lr6mTJP0ox6af39RLmclxLVavdOEBDEBgOd/zN3/zAZy3lTsO69d/36KdZZWSpJ62BD09to8GdUsxuTIAoYwwA+Ccdh+u1H9/sFWfbCuTJCXHR+uRUT30kyszFcG8GAAmI8wAOCN7TZ3mfvytFq7eo3qPocjwMN01uIumDs+WNTbK7PIAQBJhBsBpuD2G3v5yr15cvkPHqlySpOE90/Tk6F7q2q6NydUBgC/CDIBT/PiVNdpe2rBzffe0NvrlmN4a+oN2JlcFAKdHmAFCWLWrXrvKqvRtWYW2Hvp+Q9btpRWyxkbp4RHZuuOqzoqK8Ov6mgBwSQgzQAg4Xu3SzrJK7Syr1LeN/91ZVqkDx2tO237CDzP16A09lRQf3cKVXpi46EjteW602WUAMBlhBggShmHocIWzIagcrtS3330fXo5UOs/4upT4aHVPa6MuqfFaurZEkvSLMb1ZswVAwOC3FeAH/lzMzeMxdOB4jXYertROb2Cp0M6ySjlq68/4ugxrjLqltVF2WoK6p7Xx3pIbe1+qXfXeMAMAgYQwA7RS9W6P9h6r1rffVWrX4Up9+12Fdh6u1K6yKtXUuU/7mvAwqVNyXGNQaQgt2Wlt1C2tDVsMAAha/HYDTFZb59buw1WNPS0NgWVnWaWKj1Spzn363UaiIsKUlRqv7LSExt6Whl6WrNR4xUSxZxmA0EKYAVpIpbP+hEm4FdrVOJ+l5Fi1PGfYIS02KsJnSKjp1jk5TpFcYQQAkggzgF94Tkgnd7+5TsVHqnTIXnvG9okxkcpOT1D3dm2Und7G29uSYY1VONsFAMBZEWaAZlbjcmv6nzd4H6/eddR7v12CxRtYTuxpadfGorAwQgsAXAzCDNCMyhy1uudP67Rhv917bOa/9VGfjER1b5cga1zr3c+INVsABCrCDNBMNh+06+431+mQvVbW2CjZa+okST/O7ciaLQDgR8wgBJrB8s2luvWVNTpkr1W3dvFaeu9VZpcEACGDfy4Cl8AwDC34bLee+2ibDEO6JjtV8yZcoagI5r8AQEshzAAXyVXv0S/e3aj/t26/JOnOqzrp6bF9FBURrmrXmVfiBQA0L8IMcBHKq1y6b1Ghviw+pvAw6akxvTVpcBfvFUlMpgWAlkOYAS7QzrJKTX5zrfYerVYbS6TmTuiv63qkmV0WAIQswgxwAVZ9e0T/9XahKmrr1TEpVq9PulI9bAlmlwUAIY0wA5ynRV/s1dPvb5bbYyi3c5JenZir1DYWs8sCgJBHmAHOwe0x9N8fbNEb/9ojSbq5X4ae+4/L2NARAFoJwgxwFhW1dXrwnfX6dPthSdL0kT/QA9d3Z+sBAGhFCDPAGZQcq9bdb67T9u8qZIkM15zx/TT6svZmlwUAOAlhBjiNwr3luvetdTpS6VK7BIv++NMBujyzrdllAQBOgzADnOS9ogN69C/fyFXvUe/2ifrjpAHKaBtrdlkAgDMgzACNPB5DL6/Yod99slOSNLJ3ul7+ST/FW/hrAgCtGb+lAUm1dW5N//MGffDNIUnSvUO76vEbeio8nIm+ANDaEWYQ8soctbrnrUJtKDmuqIgwPfvvfTV+QKbZZQEAzhNhBiFt80G77n5znQ7Za9U2Lkqv3Jmrq7qmmF0WAOACEGYQsgq2fKeHlqxXtcutru3i9X8nXakuqfFmlwUAuECEGYQcwzD02ue7Nesf22QY0pDuqfr9hCtkjYsyuzQAwEUgzCCkuOo9+uW7m7R0XYkk6Y6BnfTMuD6Kigg3uTIAwMUizCBklFe5dN+iQn1ZfEzhYdIvx/TWXYO7sDUBAAQ4wgxCwq7DlZq8cK32HK1WG0uk5k7or+t6pJldFgCgGRBm0OpVu+rV+6l/SpK2zLxBcdEX9sf2XzuP6L8WFcpRW68ObWP1f++6Uj1sCf4oFQBgAsIMgtriL/fpqfc2qd5jKLdzkl6dmKvUNhazywIANCPCDIKS22Mo/8Oten1VsSTp5n4Zeu4/LlNMVITJlQEAmhthBkGn0lmvB99Zr0+2lUmSpo/8gR64vjsTfQEgSPn9etQDBw7ozjvvVEpKiuLi4tSvXz8VFhZ6nzcMQ88884wyMjIUGxurYcOGafPmzT7v4XQ6NXXqVKWmpio+Pl7jxo3T/v37/V06AtD+8mr9eP5qfbKtTJbIcM2b0F9Th2cTZAAgiPk1zJSXl+vqq69WVFSU/vGPf2jLli168cUX1bZtW2+b2bNna86cOZo3b57Wrl0rm82mkSNHqqKiwtsmLy9Py5Yt05IlS7Rq1SpVVlZqzJgxcrvd/iwfAebrfeW6+ff/0rbSCrVLsGjpvYM05rIMs8sCAPiZX4eZnn/+eWVmZuqNN97wHuvSpYv3vmEYevnll/Xkk0/qlltukSS9+eabSk9P1+LFi3XvvffKbrfr9ddf11tvvaURI0ZIkhYtWqTMzEytWLFCN9xwgz8/AgLEe0UH9OhfvpGr3qNe7RP1+qQBymgba3ZZAIAW4Neemffff18DBgzQrbfeqrS0NPXv31+vvfaa9/ni4mKVlpZq1KhR3mMWi0VDhw7V6tWrJUmFhYWqq6vzaZORkaGcnBxvm5M5nU45HA6fG4KTYRiaU7BDDy0pkqveoxG90vWX+wYRZAAghPg1zOzevVvz589Xdna2/vnPf+q+++7Tgw8+qD/96U+SpNLSUklSenq6z+vS09O9z5WWlio6OlpJSUlnbHOyWbNmyWq1em+ZmZnN/dHQCtTWuTX1nfX63cffSpLuvbarXp2Yq3gL89oBIJT49be+x+PRgAEDlJ+fL0nq37+/Nm/erPnz5+unP/2pt93JkzMNwzjnhM2ztZkxY4amTZvmfexwOAg0Qaasolb3/KlQG0qOKzI8TPn/3lfjr+T/MQCEIr/2zLRv3169e/f2OdarVy/t27dPkmSz2STplB6WsrIyb2+NzWaTy+VSeXn5GduczGKxKDEx0eeG4LHloEM3z/uXNpQcV9u4KL01eSBBBgBCmF/DzNVXX63t27f7HNuxY4c6d+4sScrKypLNZlNBQYH3eZfLpZUrV2rw4MGSpNzcXEVFRfm0OXTokDZt2uRtg9Dx6bYy/fiV1Tpor1XXdvF6d8rVGtQtxeyyAAAm8usw08MPP6zBgwcrPz9f48eP11dffaUFCxZowYIFkhqGl/Ly8pSfn6/s7GxlZ2crPz9fcXFxmjBhgiTJarVq8uTJmj59ulJSUpScnKxHHnlEffv29V7dhNDxwDvrZRjS1d1T9IcJubLGRZldEgDAZH4NM1deeaWWLVumGTNmaObMmcrKytLLL7+sO+64w9vmscceU01NjaZMmaLy8nINHDhQy5cvV0LC9xsBvvTSS4qMjNT48eNVU1Oj4cOHa+HChYqIYGn6UFDv9njvG4Y0YWAn/WpcH0VF+H3NRwBAAAgzDMMwuwh/czgcslqtstvtzJ8JQP/9wRb98fOGPZZm/Kin/vParqzoCwAh4HzP3/zTFq3a598e9m4WKUkTB3UmyAAAfBBm0GodrnDq4aUbFPx9hwCAS0GYQavk8Ria/ucNOlLpVHZaG7PLAQC0YoQZtEqvfb5bn+04rJiocP1m/OVmlwMAaMUIM2h1ikqO64V/NqxP9PTYPvTMAADOijCDVsVRW6ep73yteo+h0X3b6zZW9gUAnANhBq2GYRj6+d82quRYjTomxSr/lr5cuQQAOCfCDFqN/7euRH//5pAiwsP0u9v7yxrL6r4AgHMjzKBV2FlWoaff3yxJemRUD13RKcnkigAAgYIwA9PV1rn1wOL1qq3z6JrsVN17bVezSwIABBDCDEz37Adbta20QqltovXi+MsVHs48GQDA+SPMwFQfbSrVW1/slSTNGd9PaQkxJlcEAAg0hBmY5sDxGj32lw2SpHuHdtW1P2hnckUAgEBEmIEp6t0ePfTOejlq63V5Zls9MqqH2SUBAAJUpNkFIDT99uNvtW5vuRIskZp7W39FRZw5V8dFR2rPc6NbsDoAQCChZwYtbvWuI5r36U5JUv4tfdUpJc7kigAAgYwwgxZ1tNKpvCVFMgzptiszNfbyDLNLAgAEOMIMWoxhGHrkzxtUVuFU97Q2enpsH7NLAgAEAcIMWszrq4r16fbDio4M17wJ/RUbHWF2SQCAIECYQYvYuN+u5z/aJkn65Zje6mlLNLkiAECwIMzA7yqd9Zr6zteqcxu6oU+67hzYyeySAABBhDADvzIMQ79YtlF7jlarQ9tYzf6PyxUWxnYFAIDmQ5iBX/316wN6t+igIsLD9Nvb+skaF2V2SQCAIEOYgd/sOlypp97bJEl6eES2BnRJNrkiAEAwIszAL5z1bk1dvF7VLrcGdU3Rfw3rbnZJAIAgRZiBX8z6cJu2HHIoOT5aL9/WTxHhzJMBAPgHYQbNrmDLd1q4eo8k6cVbL1d6Yoy5BQEAghphBs3qkL1Gj/5lgyTp7iFZuq5nmskVAQCCHWEGzcbtMfTQkiIdr65T3w5WPXZjT7NLAgCEAMIMms3cT77VV8XHFB8dobm391d0JH+8AAD+x9kGzeLL3Uf1u4+/lSQ9++991SU13uSKAAChgjCDS1Ze5dJDS4rkMaQf53bUzf07mF0SACCEEGZwSQzD0KN/+Ualjlp1bRevX43rY3ZJAIAQQ5jBJXlz9R6t2PqdoiPCNff2/oq3RJpdEgAgxBBmcNE2H7Qr/8NtkqSf39RTfTKsJlcEAAhFhBlclCpnvaYuXi+X26MRvdI1aXAXs0sCAIQowgwuytPvb9buI1WyJcbohR9fprAwtisAAJiDMIML9u76A/pL4X6Fh0m/va2fkuKjzS4JABDCCDO4IHuOVOnJZRslSQ8Oz9bArikmVwQACHWEGZw3V71HU99ZryqXWz/MStbU67PNLgkAAMIMzt/sj7Zp4wG72sZF6be39VNEOPNkAADma7EwM2vWLIWFhSkvL897zDAMPfPMM8rIyFBsbKyGDRumzZs3+7zO6XRq6tSpSk1NVXx8vMaNG6f9+/e3VNlBp9pVry5PfKAuT3ygalf9eb/u021l+uOqYknSCz++XO2tsf4qEQCAC9IiYWbt2rVasGCBLrvsMp/js2fP1pw5czRv3jytXbtWNptNI0eOVEVFhbdNXl6eli1bpiVLlmjVqlWqrKzUmDFj5Ha7W6J0SPrOUavpf94gSbprcBeN7J1uckUAAHzP72GmsrJSd9xxh1577TUlJSV5jxuGoZdffllPPvmkbrnlFuXk5OjNN99UdXW1Fi9eLEmy2+16/fXX9eKLL2rEiBHq37+/Fi1apI0bN2rFihX+Lh2S3B5DDy8t0rEql3q3T9SMm3qaXRIAAD78Hmbuv/9+jR49WiNGjPA5XlxcrNLSUo0aNcp7zGKxaOjQoVq9erUkqbCwUHV1dT5tMjIylJOT421zOk6nUw6Hw+eGizP/f3dq9a6jiouO0NwJ/WWJjDC7JAAAfPh1I50lS5bo66+/1tq1a095rrS0VJKUnu47ZJGenq69e/d620RHR/v06DS1aXr96cyaNUu/+tWvLrX8kLduzzG9tOJbSdLMf8tRt3ZtTK4IAIBT+a1npqSkRA899JAWLVqkmJiYM7Y7eeVYwzDOuZrsudrMmDFDdrvdeyspKbmw4iF7dZ0eWlIkt8fQv/fvoP+4ooPZJQEAcFp+CzOFhYUqKytTbm6uIiMjFRkZqZUrV+p3v/udIiMjvT0yJ/ewlJWVeZ+z2WxyuVwqLy8/Y5vTsVgsSkxM9Lnh/BmGocf/+o0OHK9Rl5Q4/frmHLYrAAC0Wn4LM8OHD9fGjRtVVFTkvQ0YMEB33HGHioqK1LVrV9lsNhUUFHhf43K5tHLlSg0ePFiSlJubq6ioKJ82hw4d0qZNm7xt0PwWfblPH20uVVREmObefoXaWPw6GgkAwCXx21kqISFBOTk5Psfi4+OVkpLiPZ6Xl6f8/HxlZ2crOztb+fn5iouL04QJEyRJVqtVkydP1vTp05WSkqLk5GQ98sgj6tu37ykTitE8th5y6Nd/3yJJevzGnurb0WpyRQAAnJ2p/+R+7LHHVFNToylTpqi8vFwDBw7U8uXLlZCQ4G3z0ksvKTIyUuPHj1dNTY2GDx+uhQsXKiKCq2qaW7WrXlPfWS9XvUfX9WinyUOyzC4JAIBzCjMMwzC7CH9zOByyWq2y2+0hP3+m2lWv3k/9U5K0ZeYNiov+Ps8+8ddvtGRtidISLPrHQ9copY3FrDIBADjv8zd7M0GS9D8bDmrJ2hKFhUkv39aPIAMACBiEGajkWLV+/reNkqQHruuuwd1STa4IAIDzR5gJcXVujx54Z70qnPUa0DlJDw3PNrskAAAuCGEmxP1m+XZtKDmuxJhI/fb2/oqM4I8EACCwcOYKYf/aeUSvrtwtSZr948vVoW2syRUBAHDhCDMh7Im/NsyTmXhVZ92YYzO5GgAALg5hJoQdrXKppy1BT47uZXYpAABcNMJMCIuNitC8Cf0VE8UChACAwEWYCTFbDjm8939+U091T0s4S2sAAFo/wkyI+cu6/d77t1zRwcRKAABoHoSZEGIYhj7dXuZ9HBYWZmI1AAA0D8JMCNl0wKHvHE6zywAAoFkRZkJIwdbvzC4BAIBmR5gJISu2EGYAAMGHMBMi9pdXa8shh8KZJgMACDKEmRDx8daGib/9OyWZXAkAAM2LMBMiChqHmK7r0c7kSgAAaF6EmRDgqK3TF7uPSpKu75VmcjUAADQvwkwIWLn9sOo9hrq1i1eXlHizywEAoFkRZkJA0xDTiN7pJlcCAEDzI8wEuTq3x7vq7yjCDAAgCBFmgtxXxcdUUVuvlPho9cvkSiYAQPAhzAS5piGm4b3SFMEiMwCAIESYCWKGYWhF4xYGI3oxxAQACE6EmSC2rbRC+8trZIkM15DsVLPLAQDALwgzQaxpL6ZrslMVFx1pcjUAAPgHYSaIFTDEBAAIAYSZIFVqr9U3++0KC5OGE2YAAEGMMBOkPt7W0CvTL7Ot2iVYTK4GAAD/YSJFkPKu+ntSr0xcdKT2PDfajJIAAPALemaCUJWzXqt3Nmwsyaq/AIBgR5gJQp9/e1gut0edU+LUPa2N2eUAAOBXhJkgtLxxiGlkr3SFhbHqLwAguBFmgky926NPtzVsLMku2QCAUECYCTKFe8tVXl2ntnFRGtCZjSUBAMGPMBNkmvZiur5HmiIj+N8LAAh+nO2CiGEY31+SzRATACBEEGaCyK7DldpztFrREeG69gftzC4HAIAWQZgJIgVbGib+DuqWojYW1kMEAIQGwkwQKdhSKokhJgBAaCHMBInDFU6tLzkuSRrRK83cYgAAaEF+DTOzZs3SlVdeqYSEBKWlpenmm2/W9u3bfdoYhqFnnnlGGRkZio2N1bBhw7R582afNk6nU1OnTlVqaqri4+M1btw47d+/35+lB5xPt5XJMKS+Haxqb401uxwAAFqMX8PMypUrdf/99+uLL75QQUGB6uvrNWrUKFVVVXnbzJ49W3PmzNG8efO0du1a2Ww2jRw5UhUVFd42eXl5WrZsmZYsWaJVq1apsrJSY8aMkdvt9mf5AWX5GTaWBAAg2IUZhmG01A87fPiw0tLStHLlSl177bUyDEMZGRnKy8vT448/LqmhFyY9PV3PP/+87r33XtntdrVr105vvfWWfvKTn0iSDh48qMzMTH344Ye64YYbzvlzHQ6HrFar7Ha7EhMT/foZzVDjcqv/r5erts6jDx+8Rr0zgu8zAgBCz/mev1t0zozdbpckJScnS5KKi4tVWlqqUaNGedtYLBYNHTpUq1evliQVFhaqrq7Op01GRoZycnK8bU7mdDrlcDh8bsFs1c4jqq3zqEPbWPVqn2B2OQAAtKgWCzOGYWjatGkaMmSIcnJyJEmlpQ1X36Sn+w6NpKene58rLS1VdHS0kpKSztjmZLNmzZLVavXeMjMzm/vjtCormjaW7M3GkgCA0NNiYeaBBx7QN998o3feeeeU504+ARuGcc6T8tnazJgxQ3a73XsrKSm5+MJbOY/H0MfbmC8DAAhdLRJmpk6dqvfff1+ffvqpOnbs6D1us9kk6ZQelrKyMm9vjc1mk8vlUnl5+RnbnMxisSgxMdHnFqzWlxzXkUqXEiyR+mFWstnlAADQ4vwaZgzD0AMPPKC//e1v+uSTT5SVleXzfFZWlmw2mwoKCrzHXC6XVq5cqcGDB0uScnNzFRUV5dPm0KFD2rRpk7dNKGvaWHJYzzRFR7JsEAAg9Ph1zfv7779fixcv1nvvvaeEhARvD4zValVsbKzCwsKUl5en/Px8ZWdnKzs7W/n5+YqLi9OECRO8bSdPnqzp06crJSVFycnJeuSRR9S3b1+NGDHCn+UHhBXeS7JZKA8AEJr8Gmbmz58vSRo2bJjP8TfeeEN33XWXJOmxxx5TTU2NpkyZovLycg0cOFDLly9XQsL3V+W89NJLioyM1Pjx41VTU6Phw4dr4cKFioiI8Gf5rd6eI1X6tqxSkeFhGtaDMAMACE0tus6MWYJ1nZk/fr5b//3BVl3dPUVv332V2eUAANCsWuU6M2herPoLAABhJmCVV7m0bs8xSYQZAEBoI8wEqE+3l8ljSD1tCcpMjjO7HAAATEOYCVAFjUNMo3rTKwMACG2EmQBUW+fWyh2HJUkjCDMAgBBHmAlAa3YfVbXLrfREi3IyrGaXAwCAqQgzAWjFCVcxhYezsSQAILQRZgKMYRjeLQwYYgIAgDATcDYesOs7h1Nx0REa1DXF7HIAADAdYSbANA0xDf1BO8VEhfZ2DgAASISZgFOwtUwSC+UBANCEMBNASo5Va+shh8LDpOt7srEkAAASYSagfNw48XdAl2QlxUebXA0AAK0DYSaAFDSGmZEMMQEA4EWYCRD2mjp9ubtxY0kuyQYAwIswEyBW7jiseo+h7mltlJUab3Y5AAC0GoSZANG0seRIemUAAPBBmAkArnqP/nc7l2QDAHA6hJkA8FXxMVXU1iu1TbT6ZbY1uxwAAFoVwkwAaNqLaXjPdEWwsSQAAD4IM62cYRje+TJcxQQAwKkIM63c1kMVOnC8RjFR4RrSPdXscgAAaHUIM61c0xDTkO7tFBvNxpIAAJyMMNPKfX9JNnsxAQBwOoSZVuyQvUYbD9gVFiZd35P5MgAAnA5hphX7eGvD2jL9M9uqXYLF5GoAAGidCDOtGFcxAQBwboSZVqrSWa81u45KkkYRZgAAOCPCTCv1+Y7Dcrk96pISp27t2phdDgAArRZhppU6cWPJsDBW/QUA4EwIM61QvdujT9hYEgCA80KYaYXW7S3X8eo6tY2LUm7nJLPLAQCgVSPMtEIrGoeYru+ZpsgI/hcBAHA2nClbGcMwVNC4hcFIhpgAADgnwkwrs7OsUnuPVis6IlzX/qCd2eUAANDqEWZamaZemcHdUxRviTS5GgAAWj/CTCvjXfWXISYAAM4LYaYVKauoVVHJcUmEGQAAzhdhphX5dFuZDEO6rKNVNmuM2eUAABAQCDOtiHfVX3plAAA4bwETZv7whz8oKytLMTExys3N1eeff252Sc2qxuXW598ekcQu2QAAXIiACDNLly5VXl6ennzySa1fv17XXHONfvSjH2nfvn1ml9ZsPv/2sJz1HnVoG6uetgSzywEAIGAERJiZM2eOJk+erLvvvlu9evXSyy+/rMzMTM2fP9/s0prNiq1sLAkAwMVo9WHG5XKpsLBQo0aN8jk+atQorV692qSqmpfbY+jjrQ0bS45kiAkAgAvS6ldlO3LkiNxut9LTfU/y6enpKi0tPe1rnE6nnE6n97HD4fBrjZeqqKRcR6tcSoiJ1A+zks0uBwCAgNLqe2aanDz0YhjGGYdjZs2aJavV6r1lZma2RIkXrWBLQ6/MdT3SFMXGkgAAXJBWf+ZMTU1VRETEKb0wZWVlp/TWNJkxY4bsdrv3VlJS0hKlXrSm+TJcxQQAwIVr9WEmOjpaubm5Kigo8DleUFCgwYMHn/Y1FotFiYmJPrfWqvhIlXaWVSoyPEzDerCxJAAAF6rVz5mRpGnTpmnixIkaMGCABg0apAULFmjfvn267777zC7tkq1oXCjvqq4pSoyJMrkaAAACT0CEmZ/85Cc6evSoZs6cqUOHDiknJ0cffvihOnfubHZpl+z7jSXTTK4EAIDAFBBhRpKmTJmiKVOmmF1GszpW5dK6vcckMV8GAICL1ernzASzT7eVyWNIvdonqmNSnNnlAAAQkAgzJvJuLEmvDAAAF40wY5LaOrc++/awJHbJBgDgUhBmTLJm11FVu9yyJcYop0PrvXQcAIDWjjBjkgLvQnlpbCwJAMAlIMyYwOMx9HFTmGGICQCAS0KYMcHGA3Z953AqPjpCg7qlmF0OAAABjTBjgqa9mIb2aCdLZITJ1QAAENgIMyb4ftVfhpgAALhUhJkWVnKsWttKKxQRHqbre7KFAQAAl4ow08KahpgGdE5S27hok6sBACDwEWZaGKv+AgDQvAgzLcheXacvixs2liTMAADQPAgzLeh/d5TJ7TGUndZGnVPizS4HAICgQJhpQQwxAQDQ/AgzLcRV79HK7Q0bS44gzAAA0GwIMy3ky+KjqnDWK7WNRf06tjW7HAAAggZhpoWs8C6Ul6bwcDaWBACguRBmWoBhGFqxtUwSq/4CANDcCDMtYMshhw4cr1FMVLiGZKeaXQ4AAEGFMNMCVmxp6JW5JrudYqLYWBIAgOZEmGkBBVtLJUkjGWICAKDZEWb87JC9RpsOOBQWJl3fi40lAQBoboQZP2ua+HtFpySltrGYXA0AAMGHMONnrPoLAIB/EWb8qKK2Tmt2HZHEJdkAAPgLYcaPPttxRHVuQ1mp8erWjo0lAQDwB8KMH63Y+v0QU1gYq/4CAOAPhBk/qXd79Mk2Vv0FAMDfCDN+snZPuew1dUqKi1Ju5ySzywEAIGgRZvykaYjp+p7pimBjSQAA/IYw4wcNG0s2zZdhoTwAAPyJMHMJql316vLEB+ryxAeqdtV7j39bVqm9R6sVHRmua7LbmVghAADBjzDjB00L5V3dLUXxlkiTqwEAILgRZvygKcyMYNVfAAD8jjDTzMoqalVUclwSl2QDANASCDPN7JPGjSUv72hVemKMydUAABD8CDPNjI0lAQBoWYSZZlTtqteqnY0bSxJmAABoEYSZZvT5t0fkrPeoY1KseqQnmF0OAAAhgTDTjFZsYWNJAABamt/CzJ49ezR58mRlZWUpNjZW3bp109NPPy2Xy+XTbt++fRo7dqzi4+OVmpqqBx988JQ2Gzdu1NChQxUbG6sOHTpo5syZMgzDX6VfFLfH8G4sOZKrmAAAaDF+W9Ft27Zt8ng8evXVV9W9e3dt2rRJ99xzj6qqqvSb3/xGkuR2uzV69Gi1a9dOq1at0tGjRzVp0iQZhqG5c+dKkhwOh0aOHKnrrrtOa9eu1Y4dO3TXXXcpPj5e06dP91f5F2xDyXEdrXIpMSZSV2Ylm10OAAAhw29h5sYbb9SNN97ofdy1a1dt375d8+fP94aZ5cuXa8uWLSopKVFGRoYk6cUXX9Rdd92lZ599VomJiXr77bdVW1urhQsXymKxKCcnRzt27NCcOXM0bdq0VjOc8+n2hl6Z63qmKSqC0TsAAFpKi5517Xa7kpO/77VYs2aNcnJyvEFGkm644QY5nU4VFhZ62wwdOlQWi8WnzcGDB7Vnz57T/hyn0ymHw+Fz87dPth2WxEJ5AAC0tBYLM7t27dLcuXN13333eY+VlpYqPd335J+UlKTo6GiVlpaesU3T46Y2J5s1a5asVqv3lpmZ2Zwf5bSKj1QpKiJMQ3uwsSQAAC3pgsPMM888o7CwsLPe1q1b5/OagwcP6sYbb9Stt96qu+++2+e50w0TGYbhc/zkNk2Tf880xDRjxgzZ7XbvraSk5EI/5kW5qmuKEmOiWuRnAQCABhc8Z+aBBx7QbbfddtY2Xbp08d4/ePCgrrvuOg0aNEgLFizwaWez2fTll1/6HCsvL1ddXZ2398Vms53SA1NW1jA/5eQemyYWi8VnWKqlsOovAAAt74LDTGpqqlJTU8+r7YEDB3TdddcpNzdXb7zxhsLDfTuCBg0apGeffVaHDh1S+/btJTVMCrZYLMrNzfW2+fnPfy6Xy6Xo6Ghvm4yMDJ/Q1BoMZ74MAAAtzm9zZg4ePKhhw4YpMzNTv/nNb3T48GGVlpb69LKMGjVKvXv31sSJE7V+/Xp9/PHHeuSRR3TPPfcoMTFRkjRhwgRZLBbddddd2rRpk5YtW6b8/PxWdSWTJPW0JahD21izywAAIOT47dLs5cuXa+fOndq5c6c6duzo81zTnJeIiAh98MEHmjJliq6++mrFxsZqwoQJ3ku3JclqtaqgoED333+/BgwYoKSkJE2bNk3Tpk3zV+kX5fqeaWaXAABASAozWttSun7gcDhktVplt9u9PT7NobzKqf6/XiFJ+st9gzSgC4vlAQDQXM73/M3qbpfgi93HvPd7tWdjSQAAzECYuQSfNK76K535MnEAAOBfhJlLcNuV/l+MDwAAnB1h5hL0at98828AAMDFIcwAAICARpgBAAABjTADAAACGmEGAAAENMIMAAAIaIQZAAAQ0AgzAAAgoBFmAABAQCPMAACAgEaYAQAAAY0wAwAAAhphBgAABDTCDAAACGiEGQAAENAIMwAAIKARZgAAQEAjzAAAgIBGmAEAAAEt0uwCAllcdKT2PDfa7DIAAAhp9MwAAICARpgBAAABjTADAAACGmEGAAAENMIMAAAIaIQZAAAQ0AgzAAAgoBFmAABAQCPMAACAgEaYAQAAAY0wAwAAAhphBgAABDTCDAAACGiEGQAAENAIMwAAIKBFml1ASzAMQ5LkcDhMrgQAAJyvpvN203n8TEIizFRUVEiSMjMzTa4EAABcqIqKClmt1jM+H2acK+4EAY/Ho4MHDyohIUFhYWHN+t4Oh0OZmZkqKSlRYmJis743vsf33DL4nlsG33PL4HtuGf78ng3DUEVFhTIyMhQefuaZMSHRMxMeHq6OHTv69WckJibyl6UF8D23DL7nlsH33DL4nluGv77ns/XINGECMAAACGiEGQAAENAIM5fIYrHo6aeflsViMbuUoMb33DL4nlsG33PL4HtuGa3hew6JCcAAACB40TMDAAACGmEGAAAENMIMAAAIaIQZAAAQ0Agzl+APf/iDsrKyFBMTo9zcXH3++edmlxRUZs2apSuvvFIJCQlKS0vTzTffrO3bt5tdVtCbNWuWwsLClJeXZ3YpQenAgQO68847lZKSori4OPXr10+FhYVmlxVU6uvr9Ytf/EJZWVmKjY1V165dNXPmTHk8HrNLC2ifffaZxo4dq4yMDIWFhendd9/1ed4wDD3zzDPKyMhQbGyshg0bps2bN7dIbYSZi7R06VLl5eXpySef1Pr163XNNdfoRz/6kfbt22d2aUFj5cqVuv/++/XFF1+ooKBA9fX1GjVqlKqqqswuLWitXbtWCxYs0GWXXWZ2KUGpvLxcV199taKiovSPf/xDW7Zs0Ysvvqi2bduaXVpQef755/XKK69o3rx52rp1q2bPnq0XXnhBc+fONbu0gFZVVaXLL79c8+bNO+3zs2fP1pw5czRv3jytXbtWNptNI0eO9O6P6FcGLsoPf/hD47777vM51rNnT+OJJ54wqaLgV1ZWZkgyVq5caXYpQamiosLIzs42CgoKjKFDhxoPPfSQ2SUFnccff9wYMmSI2WUEvdGjRxs/+9nPfI7dcsstxp133mlSRcFHkrFs2TLvY4/HY9hsNuO5557zHqutrTWsVqvxyiuv+L0eemYugsvlUmFhoUaNGuVzfNSoUVq9erVJVQU/u90uSUpOTja5kuB0//33a/To0RoxYoTZpQSt999/XwMGDNCtt96qtLQ09e/fX6+99prZZQWdIUOG6OOPP9aOHTskSRs2bNCqVat00003mVxZ8CouLlZpaanPedFisWjo0KEtcl4MiY0mm9uRI0fkdruVnp7uczw9PV2lpaUmVRXcDMPQtGnTNGTIEOXk5JhdTtBZsmSJvv76a61du9bsUoLa7t27NX/+fE2bNk0///nP9dVXX+nBBx+UxWLRT3/6U7PLCxqPP/647Ha7evbsqYiICLndbj377LO6/fbbzS4taDWd+053Xty7d6/ffz5h5hKEhYX5PDYM45RjaB4PPPCAvvnmG61atcrsUoJOSUmJHnroIS1fvlwxMTFmlxPUPB6PBgwYoPz8fElS//79tXnzZs2fP58w04yWLl2qRYsWafHixerTp4+KioqUl5enjIwMTZo0yezygppZ50XCzEVITU1VRETEKb0wZWVlp6RSXLqpU6fq/fff12effaaOHTuaXU7QKSwsVFlZmXJzc73H3G63PvvsM82bN09Op1MREREmVhg82rdvr969e/sc69Wrl/7617+aVFFwevTRR/XEE0/otttukyT17dtXe/fu1axZswgzfmKz2SQ19NC0b9/ee7ylzovMmbkI0dHRys3NVUFBgc/xgoICDR482KSqgo9hGHrggQf0t7/9TZ988omysrLMLikoDR8+XBs3blRRUZH3NmDAAN1xxx0qKioiyDSjq6+++pTlBXbs2KHOnTubVFFwqq6uVni47+ktIiKCS7P9KCsrSzabzee86HK5tHLlyhY5L9Izc5GmTZumiRMnasCAARo0aJAWLFigffv26b777jO7tKBx//33a/HixXrvvfeUkJDg7QmzWq2KjY01ubrgkZCQcMo8pPj4eKWkpDA/qZk9/PDDGjx4sPLz8zV+/Hh99dVXWrBggRYsWGB2aUFl7NixevbZZ9WpUyf16dNH69ev15w5c/Szn/3M7NICWmVlpXbu3Ol9XFxcrKKiIiUnJ6tTp07Ky8tTfn6+srOzlZ2drfz8fMXFxWnChAn+L87v10sFsd///vdG586djejoaOOKK67gkuFmJum0tzfeeMPs0oIel2b7z//8z/8YOTk5hsViMXr27GksWLDA7JKCjsPhMB566CGjU6dORkxMjNG1a1fjySefNJxOp9mlBbRPP/30tL+TJ02aZBhGw+XZTz/9tGGz2QyLxWJce+21xsaNG1uktjDDMAz/RyYAAAD/YM4MAAAIaIQZAAAQ0AgzAAAgoBFmAABAQCPMAACAgEaYAQAAAY0wAwAAAhphBgAABDTCDAAACGiEGQAAENAIMwAAIKARZgAAQED7/4oGACO2L6rqAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "x_data = []\n", "y_data = []\n", "ydataerr = []\n", "times = [datetime.now()]\n", "\n", "def progress(it, metrics):\n", " times.append(datetime.now())\n", " x_data.append(it)\n", " y_data.append(metrics['eval/episode_reward'])\n", " ydataerr.append(metrics['eval/episode_reward_std'])\n", "\n", "env = envs.get_environment(\"anymal\", **env_kwargs)\n", "eval_env = envs.get_environment(\"anymal\", **env_kwargs)\n", "\n", "make_inference_fn, params, _= train_fn(environment=env,\n", " progress_fn=progress,\n", " eval_env=eval_env)\n", "\n", "plt.errorbar(x_data, y_data, yerr=ydataerr)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "demo_env = envs.training.EpisodeWrapper(env, \n", " episode_length=1000, \n", " action_repeat=1)\n", "\n", "render_rollout(\n", " jax.jit(demo_env.reset),\n", " jax.jit(demo_env.step),\n", " jax.jit(make_inference_fn(params)),\n", " demo_env,\n", " n_steps=200,\n", " camera=\"track\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A note on sample efficiency**\n", "\n", "Let's compare with PPO, again using 1e7 samples:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Text(0, 0.5, 'reward per episode')" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkcAAAGwCAYAAACjPMHLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABUDElEQVR4nO3deVhUZf8G8HuAmWEfUJRNBHdFEBdc0BSoXNJX27RcUimzRU1/oWlarwullqnla5mWCy2avaa2qG9i5kahAoKKqLghKIw7DIsMMPP8/kCmGQFlcGBY7s91zdXMOc85850TDjfPec55JEIIASIiIiICAFiYuwAiIiKi2oThiIiIiEgPwxERERGRHoYjIiIiIj0MR0RERER6GI6IiIiI9DAcEREREemxMncBdZFWq0VGRgYcHBwgkUjMXQ4RERFVghACOTk58PDwgIVFxf1DDEdVkJGRAS8vL3OXQURERFWQnp6OZs2aVbie4agKHBwcAJQcXEdHRzNXQ0RERJWhUqng5eWl+z1eEYajKig9lebo6MhwREREVMc8bEgMB2QTERER6WE4IiIiItLDcERERESkh+GIiIiISA/DEREREZEehiMiIiIiPQxHRERERHoYjoiIiIj0MBwRERER6WE4IiIiItLDcERERESkh+GIiIiISA/DEREREZEehiMiIiIiPQxHREREVCvkFxbD592d8Hl3J/ILi81WB8MRERERkR6GIyIiMlpt+QufSvD/h2kxHBERERHpYTgiIiIi0sNwREREDRZPR1F5GI6IiIiI9DAcEREREelhOCIiqkE8jUNU+1mZuwAiIiJqGNTFGtzMLcR1VQFu5KhxPUdt8N9rqgJdWyGE2epkOCIiIqIqE0JAdbcY13NKAs+NXDWuq0r/W2DwOiu/qNL7zSkohp1cWo2VV4zhiIiIiMooLNbiVt69YGPQy2PY63MjV43CYm2l9yuztEATBzlcHORoYi9HU8d//utoLcVbPyQAAOzl5osoDEdEREQNhBACOepivcBzr7cnp2wAumNELw8AOFpboamjdZnA08RBjqYO1vf+K4fCRgqJRFLuPvTH4VlYlN+mJjAcERER1XHqIo3u+f6z16G6W1y2t+fe6S21Eb08VhYSNHGQ64JNyXNrg9dNHeRwsZfDWmpZHR/NLBiOiIiIapGCIg2y8otwO68Qd/LvPfIKcTuvSPdat+7esvzCf8LRpI0JD30PB2srvYBjbRB0/glD1nCykZq1B8dcGI6IiIiqibpYowswd/IKcfvef+/ohZ/beYUGYUg/6FRFRw9HuDpWHHhc7OWwkdWfXp7qwHBERERUCeri+3p08or0wk5p+CkyeJ1XxaBjaSGBs60MjeykcLaVlTzsDF83sru3zFYGuVSCnov+BABseSMItjL+en8UPHpEVOvlFxbDd+5uAEByxEB+8ZPJ6I/V2RybjryCYtzON+zJKe3ZyVVX7aadJUFH+k/AsZXB+V7IaWRXGnykBoHHQW5V4aDl8vCGoqZVr75hfHx8cPnyZYNls2bNwkcffaR7nZaWhsmTJ+PPP/+EjY0NRo8ejaVLl0Imk9V0uUREVMPu5BUi/vIdxF6+jbjUOzhxJUu3LuK35IduXxp0nGz/CTmN7GR6r0t6d/RfO8itGuS4nbqsXoUjAIiIiMDEiRN1r+3t7XXPNRoNhgwZgiZNmiA6Ohq3bt3C+PHjIYTAypUrzVEuERFVEyEE0m/fRWzqbcRdvoO41Ns4dz23wvZPdGiKJvZyONvJdD09pT05zvfCjoM1g05DUO/CkYODA9zc3MpdFxUVheTkZKSnp8PDwwMAsGzZMoSFhWHhwoVwdHQsdzu1Wg21Wq17rVKpTF84ERE9kmKNFqczcxB3r1coNvU2rueoy7Rr1cQO3X0aoZu3M/w8HfHUimgAwMpRXXjK1sxsZVZI/WiIucuof+Ho448/xgcffAAvLy+MGDEC77zzju6UWUxMDPz8/HTBCAAGDhwItVqN+Ph4hIaGlrvPxYsXY8GCBTVSPxERVU6euhiJ6VklPUOpd3As7U6ZK72klhL4eyoQ6NMIgd7O6ObtjMb2ct16jtWh8tSrcDRt2jR07doVzs7OOHr0KGbPno1Lly5h7dq1AAClUglXV1eDbZydnSGTyaBUKivc7+zZsxEeHq57rVKp4OXlVT0fgoiIynVdVYC4y3d0YSg5UwWN1nByUgdrK3Tzdkb3e2EowMupXt2ckGpGrQ9H8+fPf2ivTWxsLAIDA/H222/rlnXq1AnOzs4YPnw4Pv74YzRu3BgAyh39L4R44FUBcrkccrm8wvVERGRaQghcuJGL2Hunx+Iv38HlW/ll2nk62SDQxxmBPo3Q3ccZbZs6cEwQPbJaH46mTJmCkSNHPrCNj49Puct79eoFADh//jwaN24MNzc3HDlyxKDNnTt3UFRUVKZHiYiIao66WIOkq9mITb2DuNQ7iL98u8zcXhIJ0N7NEd3vhaFAb2d4ONmYqWKqz2p9OHJxcYGLi0uVtk1IKLmFuru7OwAgKCgICxcuRGZmpm5ZVFQU5HI5unXrZpqCiYjoobLzi3As7Z9TZIlXssrM7G4ttUBnLycEejdCoI8zuno7w9FaaqaKqSGp9eGosmJiYnD48GGEhoZCoVAgNjYWb7/9NoYNG4bmzZsDAAYMGABfX1+MHTsWn3zyCW7fvo0ZM2Zg4sSJFV6pRkREj0YIgatZd3VXkMWl3sHZazll2jW2k/0zXsjHGR09FJBZWZihYmro6k04ksvl+PHHH7FgwQKo1Wp4e3tj4sSJmDlzpq6NpaUldu7ciUmTJqFPnz4GN4Ekqo94Z2kyB41W4IxSZRCGlKqCMu1auNghUC8MtXCxM+qu0ETVpd58U3bt2hWHDx9+aLvmzZtjx44dNVAREVHDkF9Yckl9aRhKSMsqM9WGlYUEHT0V6O5dMl6om7czmjjwQhdTqS33B6ov6k04IiKi6ieEwJU7d3H00i3dsl6L/kTxfZfU28ut0NXbGd29ndHNxxmdvZzYc0l1Bn9SiYioQln5hTh+JRuJaVk4fiULx9OzcCuv0KBNsVbAzdEagT7/nCJr7+YIS15ST3UUwxEREQEACoo0SM5U4Xh6SQhKTM9Cajn3FpJaStDO1QFJGSVTKe0J74fWTew5XojqDYYjIqIGSKsVuHgzTxeCjl/JwulMFYo0okzbFi52CGimQICXEzp7OaGDuyO0QugG+3s62TAYUb3CcERE1ABcVxXoQlBiehZOpGcjR112XrHGdjJdCArwckJAMwWcbGVl2tWXOck4kJnKw3BERFTP5KmLcfJqdkkYuvfIyC57Kb211AL+ngoENHPSBaJmzuwFImI4IiKqw4o1Wpy9loPj6dm6U2TnrufgvovHIJEAbZs6/NMj5KVAO1cHWFnyJotE92M4IiKqI0ovoz9+JUt39djJq9koKNKWaeuhsL4Xgkp6hPw8FbCX8yufqDL4L4WIqJbKzi9C4pV/To0dv5KFm7mFZdo5yK10vUEBzUrCUFNHazNUTFQ/MBwREdUCBUUanL53GX3JwOlsXLqZV6ad1FKCDu6OuhAU4OWEli52sOA9hYhMhuGIiMhMfkm8itOZOUhMN+4yemuppRmqJWo4GI6IiGqAuliDQyk38cvxq7pls7clGbSp7GX0RFS9GI6IiKpJaSDadTITe5KvlbmvUNfmTuja3BmdmzshoBkvoyeqLRiOiIhMSF2sQfS5m9h5MhN7ThkGIldHOQb4uuK7w2kAgO9f7cnJWIlqIf6rJCJ6RIXFWkSfv4GdJ5SISlYip+CfQNTUQY7B/u4Y0skd3Zo7o6BYowtHRFQ7MRwREVVBYbEWf52/iR0nMrEnWQnVAwIRryQjqlsYjoiIKqmwWIu/LtzEzhOZiDplGIiaOMgx2M8NQzp5INCbgYioLmM4IiJ6gCJNSQ/RzhOZ2F1BIBrs745An0awZCAiqhcYjoiI7lMaiHadzMTuU9eQfbdIt87FXo7B/iWBqDsDEVG9xHBERISSQPT3hVvYdSITu5OVyMrXD0QyPOXnjsH+7ujRgoGIqL5jOCKiBqtIo0XMhVvYdTITv58qG4gG+blhiL8HAxFRA8NwREQNSrFGi5iLt3RjiO7oBaLGdvcCUSd39GzRmIGIqIFiOCKieq9Yo8Xhi7ex82QGdp+6htt5/8xsrwtE906ZWVlamLFSIqoNGI6IqF4q1mhx5NJt7LjXQ6QfiBrpBaKeDEREdB+GIyKqN0oD0c6TmdidpMQtvUDkbCvFID93DPF3R6+WDEREVDGGIyKq0zRagSMXb2HnyUz8Xm4gKhlUzUBERJXFcEREdY5GK3Dk0r2rzJKUuJlrGIgGdiwZVN2rZWNIGYiIyEgMR0RUp3ywIxl7kq/jZq5at8zJVopBHUtuzBjUioGIiB4NwxER1XpCCN3zH46mAwAUNvcCUSd39GYgIiITYjgiolpv+Z4U3fNnu3ji6c4e6NPahYGIiKoFwxER1WqrD1zAuuhU3euFz/rBVsavLnOzlVkh9aMh5i6DqFrwzy4iqrV+jE3DR/87Y+4yiKiBYTgiKkd+YTF83t0Jn3d3Ir+w2NzlNEi/Jykxe9tJAMCEx3zMWwwRNSgMR0RU6/x94Sam/pAArQBeDPRCeP+25i6JiBoQhiMiqlVOXMnCxG/iUKjRYmBHVyx81g8SCSeAJaKaw3BERLXGhRu5CNsQi7xCDYJaNsaKkV14V2siqnH81iGiWiEj6y7Grj2C23mF8PdU4Ktx3WAttTR3WUTUANWZcLRw4UL07t0btra2cHJyKrdNWloahg4dCjs7O7i4uGDq1KkoLCw0aHPy5EkEBwfDxsYGnp6eiIiIMLjBHBHVvNt5hRi77ggysgvQsokdIl/uDgdrqbnLIqIGqs7cLKSwsBAjRoxAUFAQ1q1bV2a9RqPBkCFD0KRJE0RHR+PWrVsYP348hBBYuXIlAEClUqF///4IDQ1FbGwsUlJSEBYWBjs7O0yfPr2mPxIRAchVF+PlDUdx4UYe3BXW+G5CTzS2l5u7LCJqwOpMOFqwYAEAIDIystz1UVFRSE5ORnp6Ojw8PAAAy5YtQ1hYGBYuXAhHR0ds3LgRBQUFiIyMhFwuh5+fH1JSUrB8+XKEh4dXOOhTrVZDrf5nHieVSmXaD0fUQKmLNXj9uzgcv5INZ1spvpvQA55ONuYui4gauDpzWu1hYmJi4OfnpwtGADBw4ECo1WrEx8fr2gQHB0Mulxu0ycjIQGpqaoX7Xrx4MRQKhe7h5eVVbZ+DqKHQaAX+b3Mi/jp/C7YyS0S+3AOtmzqYuywiovoTjpRKJVxdXQ2WOTs7QyaTQalUVtim9HVpm/LMnj0b2dnZukd6erqJqydqWIQQeP/nk/hfkhIySwt8NTYQAV5O5i6LiAiAmcPR/PnzIZFIHviIi4ur9P7KOy0mhDBYfn+b0sHYD7qPilwuh6Ojo8GDiKruk91n8cPRdFhIgBUjO+OxNi7mLomISMesY46mTJmCkSNHPrCNj49Ppfbl5uaGI0eOGCy7c+cOioqKdL1Dbm5uZXqIrl+/DgBlepSIqHp8ffAiVu2/AABY+Kw/nvJ3N3NFRESGzBqOXFxc4OJimr8Yg4KCsHDhQmRmZsLdveTLNioqCnK5HN26ddO1mTNnDgoLCyGTyXRtPDw8Kh3CiKjqtsSlY+Gu0wCAmYPaYVSP5mauiIiorDoz5igtLQ2JiYlIS0uDRqNBYmIiEhMTkZubCwAYMGAAfH19MXbsWCQkJGDv3r2YMWMGJk6cqDsNNnr0aMjlcoSFhSEpKQnbt2/HokWLHnilGhGZRtQpJd69N5HsxL4t8GZwKzNXZB62MiukfjQEqR8Nga2szlwwTNSg1Jl/mXPnzsU333yje92lSxcAwL59+xASEgJLS0vs3LkTkyZNQp8+fWBjY4PRo0dj6dKlum0UCgX27NmDyZMnIzAwEM7OzggPD0d4eHiNfx6ihuTwxVuY8kMCNFqB4d2aYc7gDvyDhIhqrToTjiIjIyu8x1Gp5s2bY8eOHQ9s4+/vj4MHD5qwMiJ6kKSr2Xj1mzgUFmvR39cVHz3nz2BERLVanTmtRkR1z8UbuRi//ihy1cXo2aIRVo7iRLJEVPvxW4pMKr+wGD7v7oTPuzuRX1hs7nLIjJTZBRi77ihu5RXCz9MRa8cHciJZIqoTGI6IyOTu3JtI9mrWXbRwsUPkyz04kSwR1RkMR0RkUnnqYrwcGYtz13Ph5miN7yb0gAsnkiWiOoThiIhMRl2swRvfxyMxPQtO9yaSbeZsa+6yiIiMwnBERCah0QqE//c4Dp27CVuZJTaEdUcbV04kS0R1D8NRLcGBzFSXCSHw71+SsPNEJqSWEqx+qRu6NHc2d1lERFXCcEREj2z5nhRsOpIGiQT49MXO6Ne2iblLIiKqMoYjInok66IvYeWf5wEAHzzth3918jBzRUREj4bhiIiqbNuxK/hgRzIAYMaAtnipl7eZKyIienQMR0RUJX8kX8M7P50AAEx4rAUmh7Y2c0VERKbBcERERjt66TYmbzoGjVbgua6eeI8TyRJRPcJwRERGOZWRjQmRsVAXa/Fkh6b4+PlOsLBgMCKi+oPhiIgqLfVmHsavj0WOuhg9fBrh89FdIeVEskRUz/BbjYgq5ZqqAC+tO4KbuWp0cHfE2jBOJEtE9RPDERE9VFZ+IcatO4ord+7Cp7Etvn2lBxw5kSwR1VMMR0T0QPmFxXglMhZnr+WgqYMc303oiSYOnEiWiOovhiMiqlBhsRZvfn8Mx9Ky4Ghthe8m9IRXI04kS0T1G8MREZVLqxWYvuU4DqTcgI3UEhte7oF2bpxIlojqP4YjIipDCIH5v53Cb8czILWU4MuXuqKbNyeSJaKGgeGIiMr47I9z+DbmMiQSYNkLnRHSrqm5SyIiqjEMR0RkIPKvS1ix9xwAIOJpPwwL4ESyRNSwWJm7ACKqPX5OuIr5v5VMJBvevy3G1pKJZG1lVkj9aIi5yyCiBoI9R0QEANh35jpmbDkOAAjr7YO3HudEskTUMDEcERFiU2/jzY3xKNYKPNPZA3P/5cuJZImowWI4ImrgTmeq8EpkLAqKtHi8fVN8MiKAE8kSUYNWpXB06NAhvPTSSwgKCsLVq1cBAN999x2io6NNWhwRVa/Lt/Iwbv1R5BQUI9DbGV9wIlkiIuPD0datWzFw4EDY2NggISEBarUaAJCTk4NFixaZvEAiqh7XVQUYu+4obuSo0d7NAevCusNGxolkiYiMDkcffvghVq9eja+//hpS6T8TT/bu3RvHjh0zaXFEVD2y84swbv1RpN3OR/NGJRPJKmw4kSwREVCFcHT27Fn069evzHJHR0dkZWWZoiYiqkZ3CzWY8E0szihz0MRBju8n9ERTR2tzl0VEVGsYHY7c3d1x/vz5Msujo6PRsmVLkxRFRNWjSKPFpI3xiLt8B47WVvj2lR5o3pgTyRIR6TM6HL3++uuYNm0ajhw5AolEgoyMDGzcuBEzZszApEmTqqNGIjIBrVbgnS3Hse/sDVhLLbA+rDs6uDuauywiolrH6Dtkz5w5E9nZ2QgNDUVBQQH69esHuVyOGTNmYMqUKdVRIxGZwOL/ncHPiRmwspDgyzHdEOjTyNwlERHVSlWaPmThwoV47733kJycDK1WC19fX9jb25u6NiIyoY1H0gAAy14IQGh7TiRLRFSRKs+tZmtri8DAQFPWQkTVbP5QXzzd2dPcZRAR1WqVCkfPPfdcpXe4bdu2KhdDRKZ1IOWG7vmbIa0Q1qeFGashIqobKjUgW6FQ6B6Ojo7Yu3cv4uLidOvj4+Oxd+9eKBSKaiuUiIxzKiMb0/97XPd6SmgrM1ZDRFR3VCocbdiwQfdwdXXFCy+8gEuXLmHbtm3Ytm0bLl68iJEjR8LFxaXaCl24cCF69+4NW1tbODk5ldtGIpGUeaxevdqgzcmTJxEcHAwbGxt4enoiIiICQohqq5vIHJTZBZgQGYf8Qo1uGSeSJSKqHKPHHK1fvx7R0dGwtPxnmgFLS0uEh4ejd+/e+OSTT0xaYKnCwkKMGDECQUFBWLduXYXtNmzYgEGDBule6/dmqVQq9O/fH6GhoYiNjUVKSgrCwsJgZ2eH6dOnV0vdRDUtV12MVyJjoVQVoGUTO1y8kWfukoiI6hSjw1FxcTFOnz6Ndu3aGSw/ffo0tFqtyQq734IFCwAAkZGRD2zn5OQENze3ctdt3LgRBQUFiIyMhFwuh5+fH1JSUrB8+XKEh4fzL2uq84o1Wry16RiSM1VwsZdh9UtdMeDTQ+Yui4ioTjH6JpAvv/wyXnnlFSxduhTR0dGIjo7G0qVL8eqrr+Lll1+ujhqNMmXKFLi4uKB79+5YvXq1QWCLiYlBcHAw5HK5btnAgQORkZGB1NTUCvepVquhUqkMHkS1jRACETuSse/sDcitLPD1uEA0c+bdr4mIjGV0z9HSpUvh5uaGTz/9FJmZmQBKphSZOXOm2U9NffDBB3jiiSdgY2ODvXv3Yvr06bh58ybef/99AIBSqYSPj4/BNq6urrp1LVqUfyXP4sWLdT1XRLXV+r9S8W3MZUgkwGcvdkaX5s7ILyw2d1lERHWO0T1HFhYWmDlzJq5evYqsrCxkZWXh6tWrmDlzpsE4pMqYP39+uYOo9R/6V8U9zPvvv4+goCB07twZ06dPR0RERJkxUPefOisdjP2gU2qzZ89Gdna27pGenm7EpySqflGnlPhwZzIAYPZT7fGUv7uZKyIiqruqfBPIGzdu4OzZs5BIJGjXrl2VrlSbMmUKRo4c+cA29/f0GKNXr15QqVS4du0aXF1d4ebmBqVSadDm+vXrAP7pQSqPXC43OBVHVJucuJKFaZsTIQQwumdzTOzLCaCJiB6F0eEoLy8Pb731Fr799lvdeB5LS0uMGzcOK1euhK1t5cc4uLi4VOvl/wkJCbC2ttZd+h8UFIQ5c+agsLAQMpkMABAVFQUPD49HCmFE5nI16y4mfBOHu0UaBLdtgohhHXlhARHRIzL6tFp4eDgOHDiA3377TXda7ZdffsGBAweqdcxRWloaEhMTkZaWBo1Gg8TERCQmJiI3NxcA8Ntvv+Hrr79GUlISLly4gLVr1+K9997Da6+9puv1GT16NORyOcLCwpCUlITt27dj0aJFvFKN6iRVQRFe2RCLGzlqtHdzwOeju8DK0uh/0kREdB+je462bt2Kn376CSEhIbplgwcPho2NDV544QV8+eWXpqxPZ+7cufjmm290r7t06QIA2LdvH0JCQiCVSrFq1SqEh4dDq9WiZcuWiIiIwOTJk3XbKBQK7NmzB5MnT0ZgYCCcnZ0RHh6O8PDwaqmZqLoUabSYvPEYzl7LQVMHOdaHdYeDtdTcZRER1QtGh6P8/Pxyx+c0bdoU+fn5JimqPJGRkQ+8x9GgQYMMbv5YEX9/fxw8eNCElRHVLCEE5v6ShEPnbsJGaol147vDw8nG3GUREdUbRvfBBwUFYd68eSgoKNAtu3v3LhYsWICgoCCTFkdEZa05eBE/HE2HRAKsHNUF/s04pyERkSkZ3XO0YsUKDBo0CM2aNUNAQAAkEgkSExNhbW2N3bt3V0eNRHTPrpOZ+Oh/ZwAAc//liyd9K77KkoiIqsbocOTn54dz587h+++/x5kzZyCEwMiRIzFmzBjY2LBrn6i6HEu7g7d/TAQAhPX2wct9yr9pKRERPZoq3efIxsYGEydONHUtRFSBtFv5mPhNHNTFWjzZoSn+/S9fc5dERFRvGT3m6JtvvsHOnTt1r2fOnAknJyf07t0bly9fNmlxRARk5xfh5cijuJVXiI4ejlgxsgssLXjrCSKi6mJ0OFq0aJHu9FlMTAw+//xzLFmyBC4uLnj77bdNXiBRQ1ZYrMUb38fjwo08uCussT6sO+zkVb6xPRERVYLR37Lp6elo3bo1AODnn3/G8OHD8dprr6FPnz4G9z4iokcjhMDsbScRc/EW7GSWWB/WHa6O1uYui4io3jO658je3h63bt0CUDL1xpNPPgkAsLa2xt27d01bHVED9vmf57H12BVYWkjwxZiu6ODuaO6SiIgaBKN7jvr3749XX30VXbp0QUpKCoYMGQIAOHXqFOcnIzKRXxKvYtmeFADA/GEdEdKuqZkrIiJqOIzuOfriiy8QFBSEGzduYOvWrWjcuDEAID4+HqNGjTJ5gUQNzdFLt/HOlhMAgIl9W2BsL28zV0RE1LAY3XPk5OSEzz//vMzyBQsWmKQgoobs0s08vPZdHAo1Wgzq6IbZT3Uwd0lERA1OpcLRiRMn4OfnBwsLC5w4ceKBbTt16mSSwogamjt5hXh5w1Fk5RchoJkCn77YGRa8ZJ+IqMZVKhx17twZSqUSTZs2RefOnSGRSCCE0K0vfS2RSKDRaKqtWKL6Sl2swWvfxSH1Vj48nWywdnx32MgszV0WEVGDVKlwdOnSJTRp0kT3nIhMRwiBmT+dQGzqHThYWyHy5e5o4iA3d1lERA1WpcKRt7d3uc+J6NF9uicFvyRmwMpCgi/HdEMbVwdzl0RE1KBV6Va7Z8+excqVK3H69GlIJBK0b98eb731Ftq1a2fq+ojqtZ/ir+A/f54HACx61h+PtXExc0VERGT0pfw//fQT/Pz8EB8fj4CAAHTq1AnHjh2Dn58ftmzZUh01EtVLf1+4idnbSi5wmBzaCi909zJzRUREBFSh52jmzJmYPXs2IiIiDJbPmzcPs2bNwogRI0xWHFF9df56Dt74Lh5FGoF/dXLH9P7sdSUiqi2M7jlSKpUYN25cmeUvvfQSlEqlSYoiqs9u5qrxcmQsVAXF6ObtjKUjAnjJPhFRLWJ0OAoJCcGhQ4fKLI+Ojkbfvn1NUhRRfVVQpMHEb+OQfvsumjeyxVdju8Faykv2iYhqE6NPqw0bNgyzZs1CfHw8evXqBQA4fPgwtmzZggULFuDXX381aEtEJbRagfD/JiIhLQsKGyk2vNwdje15yT4RUW1jdDiaNGkSAGDVqlVYtWpVuesA8IaQRPdZsvssdp1UQmopwZqx3dCqib25SyIionIYHY60Wm111EFUr/1wNA2rD1wAACwZ3gm9WjY2c0VERFQRo8cc6SsoKDBVHUT11sGUG3j/5yQAwP892QbPdmlm5oqIiOhBjA5HGo0GH3zwATw9PWFvb4+LFy8CAP79739j3bp1Ji+QqC47q8zBpI3HoNEKPNvFE9OeaGPukoiI6CGMDkcLFy5EZGQklixZAplMplvu7++PtWvXmrQ4orrsuqoAr0TGIlddjB4tGuGj5/0hkfCSfSKi2s7ocPTtt9/iq6++wpgxY2Bp+c8lyJ06dcKZM2dMWhxRXZVfWIwJ38ThatZdtHSxw1dju0FuxUv2iYjqAqPD0dWrV9G6desyy7VaLYqKikxSFFFdptEKTNuciJNXs9HIToYNL3eHk63s4RsSEVGtYHQ46tixY7k3gdyyZQu6dOlikqKI6rJFu05jT/I1yKws8NXYbvBubGfukoiIyAhGX8o/b948jB07FlevXoVWq8W2bdtw9uxZfPvtt9ixY0d11EhUZ3wbk4p10ZcAAMtGBCDQp5GZKyIiImMZ3XM0dOhQ/Pjjj9i1axckEgnmzp2L06dP47fffkP//v2ro0aiOuHPM9cw/9dTAIB3BrbD0AAPM1dERERVYXTPEQAMHDgQAwcONHUtRHXWqYxsTNmUAK0AXghshkkhrcxdEhERVdEj3QSSiIDM7Lt4JTIW+YUa9GndGAuf5SX7RER1GcMR0SPIVRfjlcg4XFOp0aapPVaN6QapJf9ZERHVZfwWJ6qiYo0Wb206htOZKrjYy7A+rDsUNlJzl0VERI+I4YioCoQQWPBbMvadvQG5lQXWju8Or0a25i6LiIhMwKhwVFRUhJYtWyI5Obm66iGqE9ZFX8J3hy9DIgFWjOyMzl5O5i6JiIhMxKhwJJVKoVara3ywaWpqKiZMmIAWLVrAxsYGrVq1wrx581BYWGjQLi0tDUOHDoWdnR1cXFwwderUMm1OnjyJ4OBg2NjYwNPTExERERBC1OTHoTpu9yklFu46DQCY81QHDPJzN3NFRERkSkZfyv/WW2/h448/xtq1a2FlVaU7ARjtzJkz0Gq1WLNmDVq3bo2kpCRMnDgReXl5WLp0KQBAo9FgyJAhaNKkCaKjo3Hr1i2MHz8eQgisXLkSAKBSqdC/f3+EhoYiNjYWKSkpCAsLg52dHaZPn14jn4XqtuPpWZi2OQFCAGN6NserfVuYuyQiIjIxo9PNkSNHsHfvXkRFRcHf3x92doZTI2zbts1kxZUaNGgQBg0apHvdsmVLnD17Fl9++aUuHEVFRSE5ORnp6enw8Ci5+d6yZcsQFhaGhQsXwtHRERs3bkRBQQEiIyMhl8vh5+eHlJQULF++HOHh4bz8mh7oyp18TPgmDgVFWgS3bYIFwzryZ4aIqB4yekC2k5MTnn/+eQwcOBAeHh5QKBQGj5qSnZ2NRo3+mZohJiYGfn5+umAElNysUq1WIz4+XtcmODgYcrncoE1GRgZSU1MrfC+1Wg2VSmXwoIZFVVCEVyJjcTNXjfZuDvh8dBdY8ZJ9IqJ6yeieow0bNlRHHUa5cOECVq5ciWXLlumWKZVKuLq6GrRzdnaGTCaDUqnUtfHx8TFoU7qNUqlEixblnyJZvHgxFixYYMJPQHVJkUaLKZsSkHItF00d5Fgf1h0O1rxkn4iovqrSn77FxcX4448/sGbNGuTk5AAAMjIykJuba9R+5s+fD4lE8sBHXFycwTYZGRkYNGgQRowYgVdffdVgXXmnOIQQBsvvb1M6GPtBp0dmz56N7Oxs3SM9Pd2oz0l12wc7knHo3E3YyiyxPqw7PJxszF0SERFVI6N7ji5fvoxBgwYhLS0NarUa/fv3h4ODA5YsWYKCggKsXr260vuaMmUKRo4c+cA2+j09GRkZCA0NRVBQEL766iuDdm5ubjhy5IjBsjt37qCoqEjXO+Tm5qbrRSp1/fp1ACjT66RPLpcbnIqjhuWn+KuwkAArR3WBn2fNnTomIiLzMDocTZs2DYGBgTh+/DgaN26sW/7ss8+W6cl5GBcXF7i4uFSq7dWrVxEaGopu3bphw4YNsLAw7PQKCgrCwoULkZmZCXf3kkuro6KiIJfL0a1bN12bOXPmoLCwEDKZTNfGw8OjzOk2In1z/+WLJzpUHKCJiKj+MPq0WnR0NN5//31duCjl7e2Nq1evmqwwfRkZGQgJCYGXlxeWLl2KGzduQKlUGvQCDRgwAL6+vhg7diwSEhKwd+9ezJgxAxMnToSjoyMAYPTo0ZDL5QgLC0NSUhK2b9+ORYsW8Uo1MlBQpMGC3/650elLvZojrA8v2SciaiiM7jnSarXQaDRlll+5cgUODg4mKep+UVFROH/+PM6fP49mzZoZrCsdM2RpaYmdO3di0qRJ6NOnD2xsbDB69Gjdpf4AoFAosGfPHkyePBmBgYFwdnZGeHg4wsPDq6VuqnvOXcvBlE0JOHstR7ds1qD2ZqyIiIhqmtHhqH///vjss890Y34kEglyc3Mxb948DB482OQFAkBYWBjCwsIe2q558+bYsWPHA9v4+/vj4MGDJqqM6gshBH6MTcf8306hoEiLxnYy3Morubu6pQV7FYmIGhKjT6t9+umnOHDgAHx9fVFQUIDRo0fDx8cHV69exccff1wdNRJVK1VBEd76IQHvbjuJgiIt+rZxwbZJvc1dFhERmYnRPUceHh5ITEzEDz/8gGPHjkGr1WLChAkYM2YMbGx4iTPVLcfTs/DWDwlIu50PKwsJZgxsh9f6tkRBcdlTx0RE1DBUaXI0GxsbvPLKK3jllVdMXQ9RjdBqBdZFX8LHv59BsVagmbMN/jOqC7o2dzZ3aUREZGZVCkdnz57FypUrcfr0aUgkErRv3x5TpkxB+/YcuEq1381cNWZsOY79Z28AAAb7u2Hxc52gsOFdr4mIqApjjn766Sf4+fkhPj4eAQEB6NSpE44dOwZ/f39s2bKlOmokMpm/z9/E4BWHsP/sDcitLLDoWX98MborgxEREekY3XM0c+ZMzJ49GxEREQbL582bh1mzZmHEiBEmK47IVIo1Wnz2xzl8sf88hADaNLXH56O7op1b9dx+goiI6i6je46USiXGjRtXZvlLL71UZmoOotrgatZdjPzqMD7fVxKMRvXwwq9THmMwIiKichndcxQSEoJDhw6hdevWBsujo6PRt29fkxVGZAq/Jykxa+sJZN8tgoPcCoue88fQAA9zl0VERLWY0eFo2LBhmDVrFuLj49GrVy8AwOHDh7FlyxYsWLAAv/76q0FbInMoKNJg0a7T+DbmMgAgoJkCK0d1RfPGtmaujIiIajujw9GkSZMAAKtWrcKqVavKXQeU3Dm7vGlGiKrb+eu5eOuHBJzOVAEAXu/XEtMHtIPMyuizyERE1ABVaW41otpICIGf4q9g7i+ncLdIg8Z2Mix7IQAh7ZqauzQiIqpDqnSfI6LaJlddjPe3n8TPiRkAgN6tGuOzFzujqaO1mSsjIqK6huGI6ryTV7Lx1g/HkHorH5YWEoT3b4s3gltxwlgiIqoShiOqs4QQWP9XKj7632kUaQQ8nWywYmRnBPo0MndpRERUhzEcUZ10O68Q72w5jr1nrgMABnZ0xZLnA6Cw5Z2uiYjo0TAcUZ1z+OItTNucgGsqNWRWFvj3kA54qZc3JBKeRiMiokdXqXCkUqkqvUNHR8cqF0P0IMUaLVb+eR4r/zwHrQBaNrHD56O6wteDP3NERGQ6lQpHTk5Olf6rnPc2ouqQmX0X0zYn4uil2wCAFwKbYf6wjrCVsfOTiIhMq1K/Wfbt26d7npqainfffRdhYWEICgoCAMTExOCbb77B4sWLq6dKatD+SL6GGT8dR1Z+Eexkllj0nD+e7uxp7rKIiKieqlQ4Cg4O1j2PiIjA8uXLMWrUKN2yYcOGwd/fH1999RXGjx9v+iqpQVIXa/DR/85gw1+pAAB/TwVWjuoCHxc78xZGRET1mtHzKcTExCAwMLDM8sDAQBw9etQkRRFdupmH51b9rQtGrz7WAlvf7M1gRERE1c7ocOTl5YXVq1eXWb5mzRp4eXmZpChq2LYnXMG//nMIpzJUcLaVYn1YIN7/ly/nRiMiohph9GjWTz/9FM8//zx2796NXr16AQAOHz6MCxcuYOvWrSYvsKEQQuieP7vqbwR6O6ObtzO6NneGd2PbBnGZep66GP/+JQnbjl0FAPRq2QifvdgFbgpOAUJERDXH6HA0ePBgnDt3Dl9++SVOnz4NIQSefvppvPHGG+w5egTZd4t0z88qc3BWmYONR9IAAI3tZOjSvDQsOaFTMyfYyCzNVWq1OJWRjbc2JeDizTxYSID/e7ItJoe25hQgRERU44wKR0VFRRgwYADWrFmDhQsXVldNDZKTrUz3/LMXA5B0VYVjaXeQdFWFW3mF+OP0Nfxx+hoAwMpCAl8PR3Rt7oyu93qYPBTWdbJ3SQiBb2MuY+HO0yjUaOGusMaKkV3QowWnADEFW5kVUj8aYu4yiIjqFKPCkVQqRVJSUp38JVyXDOjohme6NANQcsVW0lUVjl2+g2NpdxB/+Q6u56hx4ko2TlzJRuTfqQAAV0e57jRcV29ndPRwhNyqdvcu3ckrxMytJ7AnuST0PdnBFZ8M7wRnO9lDtiQiIqo+Rp9WGzduHNatW4ePPvqoOuqh+8itLNHtXu8QUNLTcjXrLuIv30FCWhbiL99BcqYK11Rq7DqpxK6TSgCAzNICfp6OusDUzdsZTR1rz9ido5duY9rmBGRmF0BmaYE5g9tjfG8fBm8iIjI7o8NRYWEh1q5diz179iAwMBB2doaXVi9fvtxkxVFZEokEzZxt0czZVncjxPzCYpy4ko1jaXfu9TBl4XZeIY6lZeFYWhaASwAATycb3bilbt6N0N7dAVLLmr0CTKMV+GLfeXz2Rwq0AmjhYoeVo7rAz1NRo3UQERFVxOhwlJSUhK5duwIAUlJSDNbxr37zsJVZoVfLxujVsjGAkt6l1Fv5OHb5DuLvBaaz13JwNesurmbdxa/HMwAA1lILBDRzKhm3dO90XKNqPKV1TVWAaZsTcPhiyRQgz3X1RMTTfrCXcwoQIiKqPYz+raQ/lQjVThKJBC1c7NDCxQ7PdysZu5RTUITj6dm6cUsJaXegKijGkUu3ceTefGVASU9Ol+ZOutNxbV0dTHLF2L4z1zF9y3HcziuErcwSHzztp6uNiIioNuGf7A2Eg7UUj7VxwWNtXAAAWq3AhRu5urB0LC0L56/n4tLNPFy6mae715C93AqdvUp6l7o2d0KX5s5Q2Egr/b6FxVos+f0M1kaXnNrzdXfE56O7oGUTe9N/SCIiIhOoUjiKjY3Fli1bkJaWhsLCQoN127ZtM0lhVL0sLCRo4+qANq4OeLF7cwBAVn4hEtKzdFfGJaZlIVddjOjzNxF9/qZu2zZN7fWujHNCSxd7WJTTu3T5Vh5mbT2JE1eyAQBhvX0we3D7Wn8VHRERNWxGh6PNmzdj3LhxGDBgAPbs2YMBAwbg3LlzUCqVePbZZ6ujRqohTrYyhLZritB2TQGUDJ4+q8xBfNodJNwbv3T5Vj7OXc/Fueu52BybDgBQ2EhLTsU1d4avh6Nuf8NXxyBPrYGTrRSfDA9Af19Xs3wuIiIiYxgdjhYtWoRPP/0UkydPhoODA1asWIEWLVrg9ddfh7u7e3XUSGZiee9mk74ejhjbyxsAcDNXrbsi7ljaHRxPz0L23SLsP3sD+8/eMNg+T61BD59GWDGqM9wVNub4CEREREYzOhxduHABQ4aU3HFXLpcjLy8PEokEb7/9Nh5//HEsWLDA5EVS7eFiL8eAjm4Y0NENAFCk0eJ0pko3biku9TYyswsAAG+GtML0/m1hVcO3CyAiInoURoejRo0aIScnBwDg6emJpKQk+Pv7IysrC/n5+SYvkGo3qaUFOjUrme/t5T4l91zynbsbAPDW460ZjIiIqM4xOhz17dsXe/bsgb+/P1544QVMmzYNf/75J/bs2YMnnniiOmokIiIiqjFGh6PPP/8cBQUlp01mz54NqVSK6OhoPPfcc/j3v/9t8gKJiIiIapLR5zwaNWoEDw+Pko0tLDBz5kz8+uuvWL58OZydnU1eIACkpqZiwoQJaNGiBWxsbNCqVSvMmzevzG0EJBJJmcfq1asN2pw8eRLBwcGwsbGBp6cnIiIiIISolrqJiIio7jG652jMmDEICQlBcHAw2rZtWx01lXHmzBlotVqsWbMGrVu3RlJSEiZOnIi8vDwsXbrUoO2GDRswaNAg3WuF4p85u1QqFfr374/Q0FDExsYiJSUFYWFhsLOzw/Tp02vksxAREVHtZnQ4sre3x7Jly/D666/Dzc0NwcHBCA4ORkhICNq3b18dNWLQoEEGgadly5Y4e/YsvvzyyzLhyMnJCW5ubuXuZ+PGjSgoKEBkZCTkcjn8/PyQkpKC5cuXIzw8vMK54dRqNdRqte61SqUywaciIiKi2sjo02pr1qzBmTNnkJGRgeXLl0OhUGDFihXo2LFjjd7nKDs7G40aNSqzfMqUKXBxcUH37t2xevVqaLVa3bqYmBgEBwdDLpfrlg0cOBAZGRlITU2t8L0WL14MhUKhe3h5eZn0sxAREVHtUeXrrB0cHODs7AxnZ2c4OTnBysqqwh4bU7tw4QJWrlyJN954w2D5Bx98gC1btuCPP/7AyJEjMX36dCxatEi3XqlUwtXV8C7Npa+VSmWF7zd79mxkZ2frHunp6Sb8NERERFSbGH1abdasWThw4ACOHz8OPz8/9OvXD7Nnz0a/fv3g5ORk1L7mz5//0JtGxsbGIjAwUPc6IyMDgwYNwogRI/Dqq68atH3//fd1zzt37gwAiIiIMFh+/6mz0sHYFZ1SA0pudqnf20RERET1l9Hh6JNPPkGTJk0wb948PP300+jQoUOV33zKlCkYOXLkA9v4+PjonmdkZCA0NBRBQUH46quvHrr/Xr16QaVS4dq1a3B1dYWbm1uZHqLr168DQJkeJSIiImqYjA5HCQkJOHDgAPbv349ly5bB0tJSNyA7JCTEqLDk4uICFxeXSrW9evUqQkND0a1bN2zYsAEWFg8/I5iQkABra2tdj1ZQUBDmzJmDwsJCyGQyAEBUVBQ8PDwMQhgRERE1XEaPOQoICMDUqVOxbds23LhxA7t374atrS2mTp0KPz+/6qgRGRkZCAkJgZeXF5YuXYobN25AqVQa9AL99ttv+Prrr5GUlIQLFy5g7dq1eO+99/Daa6/pTomNHj0acrkcYWFhSEpKwvbt27Fo0aIHXqlGREREDYvRPUdASY/M/v37sX//fhw6dAgqlQqdO3dGaGioqesDUNK7c/78eZw/fx7NmjUzWFc6ZkgqlWLVqlUIDw+HVqtFy5YtERERgcmTJ+vaKhQK7NmzB5MnT0ZgYCCcnZ0RHh6O8PDwaqmbiIiI6h6jw5GzszNyc3MREBCAkJAQTJw4Ef369YOjo2N11AcACAsLQ1hY2APb3H8vpIr4+/vj4MGDJqqMiIiI6hujw9F3331X7WGIiIiIyFyMHnP0r3/9C46Ojjh//jx2796Nu3fvAgDnJyMiIqJ6wehwdOvWLTzxxBNo27YtBg8ejMzMTADAq6++yvnJiIiIqM4zOhy9/fbbkEqlSEtLg62trW75iy++iN9//92kxRERERHVNKPHHEVFRWH37t1lrhpr06YNLl++bLLCiIiIiMzB6J6jvLw8gx6jUjdv3uQUG0RERFTnGR2O+vXrh2+//Vb3WiKRQKvV4pNPPqm2+xwRERER1ZQqza0WEhKCuLg4FBYWYubMmTh16hRu376Nv/76qzpqJCIiIqoxRvcc+fr64sSJE+jRowf69++PvLw8PPfcc0hISECrVq2qo0YiIiKiGmNUz1FRUREGDBiANWvWYMGCBdVVExEREZHZGNVzJJVKkZSUxElaiYiIqN4y+rTauHHjsG7duuqohYiIiMjsjB6QXVhYiLVr12LPnj0IDAyEnZ2dwfrly5ebrDgiIiKimmZ0OEpKSkLXrl0BACkpKQbreLqNiIiI6jqjw9G+ffuqow4iIiKiWsHoMUdERERE9RnDEREREZEehiMiIiIiPQxHRERERHoYjoiIiIj0MBwRERER6WE4IiIiItLDcERERESkh+GIiIiISA/DEREREZEehiMiIiIiPQxHRERERHoYjoiIiIj0MBwRERER6WE4IiIiItLDcERERESkh+GIiIiISA/DEREREZEehiMiIiIiPQxHRERERHoYjoiIiIj0MBwRERER6akz4WjYsGFo3rw5rK2t4e7ujrFjxyIjI8OgTVpaGoYOHQo7Ozu4uLhg6tSpKCwsNGhz8uRJBAcHw8bGBp6enoiIiIAQoiY/ChEREdVidSYchYaG4r///S/Onj2LrVu34sKFCxg+fLhuvUajwZAhQ5CXl4fo6Ghs3rwZW7duxfTp03VtVCoV+vfvDw8PD8TGxmLlypVYunQpli9fbo6PRERERLWQlbkLqKy3335b99zb2xvvvvsunnnmGRQVFUEqlSIqKgrJyclIT0+Hh4cHAGDZsmUICwvDwoUL4ejoiI0bN6KgoACRkZGQy+Xw8/NDSkoKli9fjvDwcEgkknLfW61WQ61W616rVKrq/bBERERkNnWm50jf7du3sXHjRvTu3RtSqRQAEBMTAz8/P10wAoCBAwdCrVYjPj5e1yY4OBhyudygTUZGBlJTUyt8v8WLF0OhUOgeXl5e1fPBiIiIyOzqVDiaNWsW7Ozs0LhxY6SlpeGXX37RrVMqlXB1dTVo7+zsDJlMBqVSWWGb0telbcoze/ZsZGdn6x7p6emm+khERERUy5g1HM2fPx8SieSBj7i4OF37d955BwkJCYiKioKlpSXGjRtnMJi6vNNiQgiD5fe3Kd2+olNqACCXy+Ho6GjwICIiovrJrGOOpkyZgpEjRz6wjY+Pj+65i4sLXFxc0LZtW3To0AFeXl44fPgwgoKC4ObmhiNHjhhse+fOHRQVFel6h9zc3Mr0EF2/fh0AyvQoERERUcNk1nBUGnaqorTHp3SgdFBQEBYuXIjMzEy4u7sDAKKioiCXy9GtWzddmzlz5qCwsBAymUzXxsPDwyCEERERUcNVJ8YcHT16FJ9//jkSExNx+fJl7Nu3D6NHj0arVq0QFBQEABgwYAB8fX0xduxYJCQkYO/evZgxYwYmTpyoOw02evRoyOVyhIWFISkpCdu3b8eiRYseeKUaERERNSx1IhzZ2Nhg27ZteOKJJ9CuXTu88sor8PPzw4EDB3RXnllaWmLnzp2wtrZGnz598MILL+CZZ57B0qVLdftRKBTYs2cPrly5gsDAQEyaNAnh4eEIDw8310cjIiKiWqZO3OfI398ff/7550PbNW/eHDt27Hjovg4ePGiq0oiIiKieqRM9R0REREQ1heGIiIiISA/DEREREZEehiMiIiIiPQxHRERERHoYjoiIiIj0MBwRERER6WE4IiIiItLDcERERESkh+GIiIiISE+dmD6EqKbZyqyQ+tEQc5dBRERmwJ4jIiIiIj0MR0RERER6GI6IiIiI9DAcEREREelhOCIiIiLSw3BEREREpIfhiIiIiEgPwxERERGRHoYjIiIiIj0MR0RERER6GI6IiIiI9DAcEREREelhOCIiIiLSw3BEREREpIfhiIiIiEgPwxERERGRHoYjIiIiIj0MR0RERER6GI6IiIiI9DAcEREREelhOCIiIiLSw3BEREREpIfhiIiIiEgPwxERERGRHoYjIiIiIj0MR0RERER66kw4GjZsGJo3bw5ra2u4u7tj7NixyMjIMGgjkUjKPFavXm3Q5uTJkwgODoaNjQ08PT0REREBIURNfhQiIiKqxazMXUBlhYaGYs6cOXB3d8fVq1cxY8YMDB8+HH///bdBuw0bNmDQoEG61wqFQvdcpVKhf//+CA0NRWxsLFJSUhAWFgY7OztMnz69xj4LERER1V51Jhy9/fbbuufe3t5499138cwzz6CoqAhSqVS3zsnJCW5ubuXuY+PGjSgoKEBkZCTkcjn8/PyQkpKC5cuXIzw8HBKJpNo/R0VsZVZI/WiI2d6fiIiIStSZ02r6bt++jY0bN6J3794GwQgApkyZAhcXF3Tv3h2rV6+GVqvVrYuJiUFwcDDkcrlu2cCBA5GRkYHU1NQK30+tVkOlUhk8iIiIqH6qU+Fo1qxZsLOzQ+PGjZGWloZffvnFYP0HH3yALVu24I8//sDIkSMxffp0LFq0SLdeqVTC1dXVYJvS10qlssL3Xbx4MRQKhe7h5eVlwk9FREREtYlZw9H8+fPLHUSt/4iLi9O1f+edd5CQkICoqChYWlpi3LhxBoOp33//fQQFBaFz586YPn06IiIi8Mknnxi85/2nzkq3f9AptdmzZyM7O1v3SE9PN8XHJyIiolrIrGOOpkyZgpEjRz6wjY+Pj+65i4sLXFxc0LZtW3To0AFeXl44fPgwgoKCyt22V69eUKlUuHbtGlxdXeHm5lamh+j69esAUKZHSZ9cLjc4FUdERET1l1nDUWnYqYrSHh+1Wl1hm4SEBFhbW8PJyQkAEBQUhDlz5qCwsBAymQwAEBUVBQ8PD4MQRkRERA1Xnbha7ejRozh69Cgee+wxODs74+LFi5g7dy5atWql6zX67bffoFQqERQUBBsbG+zbtw/vvfceXnvtNV2vz+jRo7FgwQKEhYVhzpw5OHfuHBYtWoS5c+ea9Uo1IiIiqj3qRDiysbHBtm3bMG/ePOTl5cHd3R2DBg3C5s2bdcFHKpVi1apVCA8Ph1arRcuWLREREYHJkyfr9qNQKLBnzx5MnjwZgYGBcHZ2Rnh4OMLDw8310YiIiKiWkQjeHtpoKpUKCoUC2dnZcHR0NHc5tUp+YTF85+4GACRHDIStrE7kbyIiagAq+/u7Tl3KT0RERFTdGI6IiIiI9DAcEREREelhOCIiIiLSw3BEREREpIfhiIiIiEgPwxERERGRHoYjIiIiIj0MR0RERER6GI6IiIiI9DAcEREREelhOCIiIiLSw3BEREREpIfhiIiIiEgPwxERERGRHoYjIiIiIj0MR0RERER6GI6IiIiI9FiZuwCqX2xlVkj9aIi5yyAiIqoy9hwRERER6WE4IiIiItLDcERERESkh+GIiIiISA/DEREREZEehiMiIiIiPQxHRERERHoYjoiIiIj0MBwRERER6WE4IiIiItLDcERERESkh+GIiIiISA/DEREREZEehiMiIiIiPQxHRERERHqszF1AXSSEAACoVCozV0JERESVVfp7u/T3eEUYjqogJycHAODl5WXmSoiIiMhYOTk5UCgUFa6XiIfFJypDq9UiIyMDDg4OkEgkJtuvSqWCl5cX0tPT4ejoaLL91mc8Zsbh8TIOj5fxeMyMw+NlvEc5ZkII5OTkwMPDAxYWFY8sYs9RFVhYWKBZs2bVtn9HR0f+IzESj5lxeLyMw+NlPB4z4/B4Ga+qx+xBPUalOCCbiIiISA/DEREREZEehqNaRC6XY968eZDL5eYupc7gMTMOj5dxeLyMx2NmHB4v49XEMeOAbCIiIiI97DkiIiIi0sNwRERERKSH4YiIiIhID8MRERERkR6Goxq2atUqtGjRAtbW1ujWrRsOHTr0wPYHDhxAt27dYG1tjZYtW2L16tU1VGntYMzx2rZtG/r3748mTZrA0dERQUFB2L17dw1WWzsY+zNW6q+//oKVlRU6d+5cvQXWMsYeL7Vajffeew/e3t6Qy+Vo1aoV1q9fX0PV1g7GHrONGzciICAAtra2cHd3x8svv4xbt27VULXmdfDgQQwdOhQeHh6QSCT4+eefH7pNQ/7eN/Z4Vdv3vqAas3nzZiGVSsXXX38tkpOTxbRp04SdnZ24fPlyue0vXrwobG1txbRp00RycrL4+uuvhVQqFT/99FMNV24exh6vadOmiY8//lgcPXpUpKSkiNmzZwupVCqOHTtWw5Wbj7HHrFRWVpZo2bKlGDBggAgICKiZYmuBqhyvYcOGiZ49e4o9e/aIS5cuiSNHjoi//vqrBqs2L2OP2aFDh4SFhYVYsWKFuHjxojh06JDo2LGjeOaZZ2q4cvPYtWuXeO+998TWrVsFALF9+/YHtm/o3/vGHq/q+t5nOKpBPXr0EG+88YbBsvbt24t333233PYzZ84U7du3N1j2+uuvi169elVbjbWJscerPL6+vmLBggWmLq3Wquoxe/HFF8X7778v5s2b16DCkbHH63//+59QKBTi1q1bNVFerWTsMfvkk09Ey5YtDZb95z//Ec2aNau2Gmuryvyyb+jf+/oqc7zKY4rvfZ5WqyGFhYWIj4/HgAEDDJYPGDAAf//9d7nbxMTElGk/cOBAxMXFoaioqNpqrQ2qcrzup9VqkZOTg0aNGlVHibVOVY/Zhg0bcOHCBcybN6+6S6xVqnK8fv31VwQGBmLJkiXw9PRE27ZtMWPGDNy9e7cmSja7qhyz3r1748qVK9i1axeEELh27Rp++uknDBkypCZKrnMa8ve+KZjqe58Tz9aQmzdvQqPRwNXV1WC5q6srlEpludsolcpy2xcXF+PmzZtwd3evtnrNrSrH637Lli1DXl4eXnjhheoosdapyjE7d+4c3n33XRw6dAhWVg3r66Aqx+vixYuIjo6GtbU1tm/fjps3b2LSpEm4fft2gxh3VJVj1rt3b2zcuBEvvvgiCgoKUFxcjGHDhmHlypU1UXKd05C/903BVN/77DmqYRKJxOC1EKLMsoe1L295fWXs8Sr1ww8/YP78+fjxxx/RtGnT6iqvVqrsMdNoNBg9ejQWLFiAtm3b1lR5tY4xP2NarRYSiQQbN25Ejx49MHjwYCxfvhyRkZENpvcIMO6YJScnY+rUqZg7dy7i4+Px+++/49KlS3jjjTdqotQ6qaF/71eVKb/3G9afimbk4uICS0vLMn9dXb9+vcxfCaXc3NzKbW9lZYXGjRtXW621QVWOV6kff/wREyZMwJYtW/Dkk09WZ5m1irHHLCcnB3FxcUhISMCUKVMAlPzyF0LAysoKUVFRePzxx2ukdnOoys+Yu7s7PD09oVAodMs6dOgAIQSuXLmCNm3aVGvN5laVY7Z48WL06dMH77zzDgCgU6dOsLOzQ9++ffHhhx+yJ+Q+Dfl7/1GY+nufPUc1RCaToVu3btizZ4/B8j179qB3797lbhMUFFSmfVRUFAIDAyGVSqut1tqgKscLKPnLISwsDJs2bWpwYxqMPWaOjo44efIkEhMTdY833ngD7dq1Q2JiInr27FlTpZtFVX7G+vTpg4yMDOTm5uqWpaSkwMLCAs2aNavWemuDqhyz/Px8WFgY/qqxtLQE8E+PCP2jIX/vV1W1fO8/0nBuMkrpJbDr1q0TycnJ4v/+7/+EnZ2dSE1NFUII8e6774qxY8fq2pde0vn222+L5ORksW7dugZ1Saexx2vTpk3CyspKfPHFFyIzM1P3yMrKMtdHqHHGHrP7NbSr1Yw9Xjk5OaJZs2Zi+PDh4tSpU+LAgQOiTZs24tVXXzXXR6hxxh6zDRs2CCsrK7Fq1Spx4cIFER0dLQIDA0WPHj3M9RFqVE5OjkhISBAJCQkCgFi+fLlISEjQ3fqA3/uGjD1e1fW9z3BUw7744gvh7e0tZDKZ6Nq1qzhw4IBu3fjx40VwcLBB+/3794suXboImUwmfHx8xJdfflnDFZuXMccrODhYACjzGD9+fM0XbkbG/ozpa2jhSAjjj9fp06fFk08+KWxsbESzZs1EeHi4yM/Pr+GqzcvYY/af//xH+Pr6ChsbG+Hu7i7GjBkjrly5UsNVm8e+ffse+L3E731Dxh6v6vrelwjBfk0iIiKiUhxzRERERKSH4YiIiIhID8MRERERkR6GIyIiIiI9DEdEREREehiOiIiIiPQwHBERERHpYTgiIiKiWuHgwYMYOnQoPDw8IJFI8PPPPxu1/fz58yGRSMo87OzsjNoPwxER1UqpqamQSCRITEw0dylEVEPy8vIQEBCAzz//vErbz5gxA5mZmQYPX19fjBgxwqj9MBwRNXA3btyAVCpFfn4+iouLYWdnh7S0NHOXBS8vL2RmZsLPz8/cpVSrkJAQ/N///Z/Z90FUGzz11FP48MMP8dxzz5W7vrCwEDNnzoSnpyfs7OzQs2dP7N+/X7fe3t4ebm5uuse1a9eQnJyMCRMmGFWH1aN8CCKq+2JiYtC5c2fY2triyJEjaNSoEZo3b27usmBpaQk3N7cK1wshoNFoYGXFrzGihuLll19GamoqNm/eDA8PD2zfvh2DBg3CyZMn0aZNmzLt165di7Zt26Jv375GvQ97jogauL///ht9+vQBAERHR+ueP8yGDRvQoUMHWFtbo3379li1apVuXekpsW3btiE0NBS2trYICAhATEwMACA7Oxs2Njb4/fffDfa5bds22NnZITc3t8xptf3790MikWD37t0IDAyEXC7HoUOHoFarMXXqVDRt2hTW1tZ47LHHEBsbq9tn6XZ79+5FYGAgbG1t0bt3b5w9e1bXZv78+ejcuTPWr1+P5s2bw97eHm+++SY0Gg2WLFkCNzc3NG3aFAsXLjSoNzs7G6+99hqaNm0KR0dHPP744zh+/HiZ/X733Xfw8fGBQqHAyJEjkZOTAwAICwvDgQMHsGLFCt3YiNTU1HKP96pVq9CmTRtYW1vD1dUVw4cPf+g+kpOTMXjwYNjb28PV1RVjx47FzZs3dfsMCQnBlClTMGXKFDg5OaFx48Z4//33oT/lZkXvS1TTLly4gB9++AFbtmxB37590apVK8yYMQOPPfYYNmzYUKa9Wq3Gxo0bje41AgA80rS1RFQnXb58WSgUCqFQKIRUKhXW1tZCoVAImUwm5HK5UCgU4s0336xw+6+++kq4u7uLrVu3iosXL4qtW7eKRo0aicjISCGEEJcuXRIARPv27cWOHTvE2bNnxfDhw4W3t7coKioSQgjx/PPPi5deeslgv88//7wYNWqUwT4SEhKEEP/M1t2pUycRFRUlzp8/L27evCmmTp0qPDw8xK5du8SpU6fE+PHjhbOzs7h165bBdj179hT79+8Xp06dEn379hW9e/fWve+8efOEvb29GD58uDh16pT49ddfhUwmEwMHDhRvvfWWOHPmjFi/fr0AIGJiYoQQQmi1WtGnTx8xdOhQERsbK1JSUsT06dNF48aNde9dut/nnntOnDx5Uhw8eFC4ubmJOXPmCCGEyMrKEkFBQWLixIkiMzNTZGZmiuLi4jLHOzY2VlhaWopNmzaJ1NRUcezYMbFixYoH7iMjI0O4uLiI2bNni9OnT4tjx46J/v37i9DQUN1+g4ODhb29vZg2bZo4c+aM+P7774Wtra346quvHvq+RNUNgNi+fbvu9X//+18BQNjZ2Rk8rKysxAsvvFBm+02bNgkrKyuRmZlp/Hs/SuFEVDcVFRWJS5cuiePHjwupVCoSExPF+fPnhb29vThw4IC4dOmSuHHjRoXbe3l5iU2bNhks++CDD0RQUJAQ4p9gs3btWt36U6dOCQDi9OnTQgghtm3bJuzt7UVeXp4QQojs7GxhbW0tdu7cabCP+8PRzz//rNtnbm6ukEqlYuPGjbplhYWFwsPDQyxZssRguz/++EPXZufOnQKAuHv3rhCiJMTY2toKlUqlazNw4EDh4+MjNBqNblm7du3E4sWLhRBC7N27Vzg6OoqCggKD49CqVSuxZs2aCvf7zjvviJ49e+peBwcHi2nTplVwpEts3bpVODo6GuxHX3n7+Pe//y0GDBhgsCw9PV0AEGfPntVt16FDB6HVanVtZs2aJTp06FCp9yWqTveHo82bNwtLS0tx5swZce7cOYNHeQHo8ccfF88880yV3pun1YgaICsrK/j4+ODMmTPo3r07AgICoFQq4erqin79+sHHxwcuLi7lbnvjxg2kp6djwoQJsLe31z0+/PBDXLhwwaBtp06ddM/d3d0BANevXwcADBkyBFZWVvj1118BAFu3boWDgwMGDBjwwNoDAwN1zy9cuICioiKDU4FSqRQ9evTA6dOnK10LAPj4+MDBwUH32tXVFb6+vrCwsDBYVrpNfHw8cnNz0bhxY4PjcOnSJYPjcP9+3d3dDd63Mvr37w9vb2+0bNkSY8eOxcaNG5Gfn//AbeLj47Fv3z6D2tq3bw8ABvX16tULEolE9zooKAjnzp2DRqOp0vsSVZcuXbpAo9Hg+vXraN26tcHj/vGJly5dwr59+6p2Sg0ckE3UIHXs2BGXL19GUVERtFot7O3tUVxcjOLiYtjb28Pb2xunTp0qd1utVgsA+Prrr9GzZ0+DdZaWlgavpVKp7nnpL+DS7WUyGYYPH45NmzZh5MiR2LRpE1588cWHDrDWv1+JuDc2Rv+Xe+ny+5c9qJb715e2KW9Z6TZarRbu7u4GV8qUcnJyeuB+9d+3MhwcHHDs2DHs378fUVFRmDt3LubPn4/Y2FiD99Kn1WoxdOhQfPzxx2XWlYbD6nhfokeRm5uL8+fP615funQJiYmJaNSoEdq2bYsxY8Zg3LhxWLZsGbp06YKbN2/izz//hL+/PwYPHqzbbv369XB3d8dTTz1VpTrYc0TUAO3atQuJiYlwc3PD999/j8TERPj5+eGzzz5DYmIidu3aVeG2rq6u8PT0xMWLF8v89daiRQuj6hgzZgx+//13nDp1Cvv27cOYMWOM2r5169aQyWSIjo7WLSsqKkJcXBw6dOhg1L6M1bVrVyiVSlhZWZU5DhX1upVHJpNBo9E8tJ2VlRWefPJJLFmyBCdOnEBqair+/PPPCvfRtWtXnDp1Cj4+PmXq0w+Yhw8fNtju8OHDaNOmjS7oPuh9iUwtLi4OXbp0QZcuXQAA4eHh6NKlC+bOnQug5EKQcePGYfr06WjXrh2GDRuGI0eOwMvLS7cPrVaLyMhIhIWFlfmDrbLYc0TUAHl7e0OpVOLatWt4+umnYWFhgeTkZDz33HPw8PB46Pbz58/H1KlT4ejoiKeeegpqtRpxcXG4c+cOwsPDK11HcHAwXF1dMWbMGPj4+KBXr15GfQ47Ozu8+eabeOedd3S3IFiyZAny8/Or3J1eWU8++SSCgoLwzDPP4OOPP0a7du2QkZGBXbt24ZlnnjE4/fcgPj4+OHLkCFJTU2Fvb49GjRoZnMoDgB07duDixYvo168fnJ2dsWvXLmi1WrRr167CfUyePBlff/01Ro0ahXfeeQcuLi44f/48Nm/ejK+//lr3SyM9PR3h4eF4/fXXcezYMaxcuRLLli2r1PsSmVpISIjB1ZL3k0qlWLBgARYsWFBhGwsLC6Snpz9SHew5Imqg9u/fj+7du8Pa2hpHjhyBp6dnpYIRALz66qtYu3YtIiMj4e/vj+DgYERGRhrdcySRSDBq1CgcP37c6F6jUh999BGef/55jB07Fl27dsX58+exe/duODs7V2l/lSWRSLBr1y7069cPr7zyCtq2bYuRI0ciNTUVrq6uld7PjBkzYGlpCV9fXzRp0qTcG3A6OTlh27ZtePzxx9GhQwesXr0aP/zwAzp27FjhPjw8PPDXX39Bo9Fg4MCB8PPzw7Rp06BQKAzC17hx43D37l306NEDkydPxltvvYXXXnutUu9LVF9JxIMiGhER1VshISHo3LkzPvvsM3OXQlSrsOeIiIiISA/DEREREZEenlYjIiIi0sOeIyIiIiI9DEdEREREehiOiIiIiPQwHBERERHpYTgiIiIi0sNwRERERKSH4YiIiIhID8MRERERkZ7/B/hRl3NJjltmAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "train_fn = functools.partial(\n", " ppo.train, num_timesteps=10_000_000, num_evals=10, reward_scaling=0.1,\n", " episode_length=1000, normalize_observations=True, action_repeat=1,\n", " unroll_length=10, num_minibatches=32, num_updates_per_batch=8,\n", " discounting=0.97, learning_rate=3e-4, entropy_cost=1e-3, num_envs=1024,\n", " batch_size=1024, seed=0)\n", "\n", "x_data = []\n", "y_data = []\n", "ydataerr = []\n", "\n", "env = envs.get_environment(\"anymal\", **env_kwargs)\n", "\n", "def progress(num_steps, metrics):\n", " x_data.append(num_steps)\n", " y_data.append(metrics['eval/episode_reward'])\n", " ydataerr.append(metrics['eval/episode_reward_std'])\n", "\n", "make_inference_fn, params, _= train_fn(environment=env, progress_fn=progress)\n", "\n", "plt.errorbar(x_data, y_data, yerr=ydataerr)\n", "plt.xlabel('# environment steps')\n", "plt.ylabel('reward per episode')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that PPO struggles to learn locomotion in this setup, even with over 10x the number of simulator steps. \n", "\n", "Rather than indicating a shortcoming of PPO, this study demonstrates a policy-learning setup that effectively leverages FoPG methods. FoPG's perform well here because it involves learning small, accurate perturbations from a good baseline - optimizing to the local minima in a deep valley. FoPG's are precise enough to guide these perturbations using nuanced reward signals, such as our foot placement spline.\n", "\n", "In contrast, RL algorithms such as PPO benefit from [policy-learning setups](https://colab.research.google.com/github/google-deepmind/mujoco/blob/main/mjx/tutorial.ipynb) that have less structured rewards. Unlike FoPG methods, they [benefit greatly](https://www.science.org/doi/abs/10.1126/scirobotics.adg1462) from sparse, non-differentiable rewards such as a large penalty for falling." ] } ], "metadata": { "kernelspec": { "display_name": "mujoco", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 2 }