diff --git a/mjx/mujoco/mjx/test_data/anymal_c/assets/anymal_c.xml b/mjx/mujoco/mjx/test_data/anymal_c/assets/anymal_c.xml index 205e08ad..c674037d 100644 --- a/mjx/mujoco/mjx/test_data/anymal_c/assets/anymal_c.xml +++ b/mjx/mujoco/mjx/test_data/anymal_c/assets/anymal_c.xml @@ -1,5 +1,5 @@ - + diff --git a/mjx/training_apg.ipynb b/mjx/training_apg.ipynb index 65133668..9a22a8bf 100644 --- a/mjx/training_apg.ipynb +++ b/mjx/training_apg.ipynb @@ -4,28 +4,30 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### Policy learning and Policy Gradients\n", + "## Policy learning and Policy Gradients\n", "\n", - "This is a recap of policy learning, and how it differs when one assumes a stochastic versus a deterministic model. If these concepts are unfamiliar to you, there are many great resources [online](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html). This recap contextualizes how we can use MJX's differentiability for policy learning.\n", + "This is a recap of policy learning contextualizes how we can use MJX's differentiability for policy learning. If the below concepts are unfamiliar, there are many great resources [online](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html)! \n", "\n", - "The goal of policy learning is to learn a control policy $\\pi$ which outputs actions $a_t \\sim \\pi(\\cdot| x_t, \\theta)$ maximizing the total rewards $\\sum r_t$, where $r_t$ is shorthand for a reward function evaluated at the state and action of time t: $r_t = r(x_t, a_t)$. $\\theta$ are the parameters of the policy. In the common case that the policy is a neural network, $\\theta$ would be the weights. **Policy gradient methods** involve estimating the gradient of the policy with respect to the weights, and using this value in a first-order optimization algorithm such as Gradient Descent or [Adam](https://arxiv.org/abs/1412.6980).\n", + "The goal of policy learning is to find a control policy $\\pi$ which outputs actions $a_t \\sim \\pi(\\cdot| x_t, \\theta)$ maximizing the total rewards $\\sum r_t$ over some time period, where $r_t$ is shorthand for a reward function evaluated at the state and action of time t:\n", "\n", - "How you estimate the policy gradient depends on what state transition model you assume. \n", + "$$r_t = r(x_t, a_t)$$\n", + "\n", + "$\\theta$ are the parameters of the policy; weights in the common case that the policy is a neural network. **Policy gradient methods** involve estimating the gradient of the rewards with respect to the weights, and using this value in a first-order optimization algorithm such as Gradient Descent or [Adam](https://arxiv.org/abs/1412.6980). How we estimate the policy gradient depends on what state transition model we assume. \n", "\n", "#### Zeroth-Order Policy Gradients (ZoPG)\n", "\n", "Referring to mjx.step as the simulation function f, we borrowing some [terminology](https://arxiv.org/abs/2202.00817) to differentiate between zeroth-order gradients, which only depend on values of f, and first-order gradients, which depend on its jacobian.\n", "\n", - "Reinforcement learning algorithms such as the standard [PPO](https://github.com/google/brax/blob/main/brax/training/agents/ppo/train.py) assume a stochastic state transition model $x_{t+1} \\sim P(\\cdot | x_t, a_t)$. This leads to a ZoPG of the form:\n", + "Reinforcement learning (RL) algorithms such as the standard [PPO](https://github.com/google/brax/blob/main/brax/training/agents/ppo/train.py) assume the stochastic state transition model $x_{t+1} \\sim P(\\cdot | x_t, a_t)$. This leads to a ZoPG of the form:\n", "\n", "$$\n", "\\nabla_\\theta J(\\pi_\\theta) = \\mathbb{E}_{\\tau \\sim \\pi_\\theta}\\left[ \\sum \\nabla_\\theta \\log\\pi_\\theta (a_t | s_t) R(\\tau) \\right]\n", "$$\n", "\n", - "Despite this method's popularity and extensive research into its refinement, a fundamental shortcoming is that the gradient has high variance. This allows the optimizer to thoroughly explore the space of policies, leading to the robust and often surprisingly good policies that have been achieved. However, the variance comes at the cost of requiring many samples $(x_t, a_t)$ to converge.\n", + "Where $R(\\tau)$ is some function depending on the rollout $\\tau = \\{x_t, a_t\\}_{t=0}^{T}$. Despite this method's popularity and extensive research into its refinement, a fundamental property is that the gradient has high variance. This allows the optimizer to thoroughly explore the space of policies, leading to the robust and often surprisingly good policies that have been achieved. However, the variance comes at the cost of requiring many samples $(x_t, a_t)$ to converge.\n", "\n", "#### First-Order Policy Gradients (FoPG)\n", - "On the other hand, if you assume a deterministic state transition model $x_{t+1} = f(x_t, a_t)$, you end up with the first-order policy gradient. Unlike ZoPG methods, which models the state evolution as a probabilistic black box, the FoPG explicitly contains the jacobians of the simulation function f. For example, let's look at the gradient of $r_t$, in the case that it only depends on state.\n", + "On the other hand, if you assume a deterministic state transition model $x_{t+1} = f(x_t, a_t)$, you end up with the first-order policy gradient. Other common names include Analytical Policy Gradients (APG) and Backpropogation through Time (BPTT). Unlike ZoPG methods, which model the state evolution as a probabilistic black box, the FoPG explicitly contains the jacobians of the simulation function f. For example, let's look at the gradient of $r_t$, in the case that it only depends on state.\n", "$$\n", "\\frac{\\partial r_t}{\\partial \\theta} = \\frac{\\partial r_t}{\\partial x_t}\\frac{\\partial x_t}{\\partial \\theta} \n", "$$\n", @@ -34,48 +36,1205 @@ "\\frac{\\partial x_t}{\\partial \\theta} = \\textcolor{Navy}{\\frac{\\partial f(x_t, a_t)}{\\partial x_{t-1}}}\\frac{\\partial x_{t-1}}{\\partial \\theta} + \\textcolor{Navy}{\\frac{\\partial f(x_t, a_t)}{\\partial a_{t-1}}} \\frac{\\partial a_{t-1}}{\\partial \\theta}\n", "$$\n", "\n", - "The navy-colored terms in the above expression are enabled by MJX's differentiability and are the key difference between FoPG's and ZoPG's. An important consideration is what these jacobians look like near contact points. To see why certain gradients within the jacobian can be pathological, imagine a hard sphere falling toward a block of marble. How does its velocity change with respect to distance ($\\frac{\\partial \\dot{z}_t}{\\partial z_t}$, for $x_t$ = [$z_t, \\dot{z}_t$]), the instant before it touches the ground? This is the case of an **uninformative gradient**, due to **hard contact**. In practice however, the default contact settings in Mujoco are sufficiently soft for learning via FoPG's. Soft contacts would resolve the above scenario by modelling the ground as applying an increasing force on the ball as it penetrates it.\n", + "The navy-colored terms in the above expression are enabled by MJX's differentiability and are the key difference between FoPG's and ZoPG's. An important consideration is what these jacobians look like near contact points. To see why certain gradients within the jacobian can be pathological, imagine a hard sphere falling toward a block of marble. How does its velocity change with respect to distance ($\\frac{\\partial \\dot{z}_t}{\\partial z_t}$), the instant before it touches the ground? This is the case of an **uninformative gradient**, due to **hard contact**. Fortunately, the default contact settings in Mujoco are sufficiently [soft](https://mujoco.readthedocs.io/en/stable/computation/index.html#soft-contact-model) for learning via FoPG's. With soft contacts, the ground applies an increasing force on the ball as it penetrates it, unlike rigid contacts, which instantly provide enough force for deflection.\n", + "\n", + "A helpful way to think about FoPG's is via the chain rule, as illustrated below for how $r_2$ influences the policy gradient, again for the case that the reward does not depend on action:\n", "\n", - "A helpful way to think about FoPG's is via the chain rule, as illustrated below for how $r_2$ influences the parameter update, again for the case that the reward does not depend on action:\n", "\"drawing\"\n", "\n", - "Note that there three distinct gradient chains in this example. The red pathway does not use the simulator's differentiability. The blue path is the most intuitive usage of this feature, and captures how actions affect downstream rewards. The least intuitive may be the green chain, which shows how the reward depends on how actions depend on previous actions - experience shows that blocking this pathway via jax.lax.stop_grad can badly hinder policy learning. As the length of $x_t$ backbone increases, [gradient explosion](https://arxiv.org/abs/2111.05803) becomes a crucial consideration. In practice, this can be resolved via decaying downstream gradients or periodically truncating the gradient.\n", + "Note that there three distinct gradient chains in this example. The red pathway considers how the immediately prior action affected the state. The blue path explains the name \"Backpropogation through Time\", capturing how actions affect downstream rewards. The least intuitive may be the green chain, which shows how the reward depends on how actions depend on previous actions - experience shows that blocking this pathway via jax.lax.stop_grad can badly hinder policy learning. As the length of $x_t$ backbone increases, [gradient explosion](https://arxiv.org/abs/2111.05803) becomes a crucial consideration. In practice, this can be resolved via decaying downstream gradients or periodically truncating the gradient.\n", "\n", "**The Sharp bits of FoPG's**\n", "\n", - "While FoPG's have been shown to be very sample efficient, especially as the [dimension of the state space increases](https://arxiv.org/abs/2204.07137), they can still struggle with wall-clock time. Because the gradients have low variance, they do not benefit significantly from massive parallelization of data collection - unlike [RL](https://arxiv.org/abs/2109.11978). Additionally, the policy gradient is typically calculated via autodifferentiation. This can be 3-5x slower than unrolling the simulation forward, and memory intensive, with memory requirements scaling with $O((m+n) \\cdot m \\cdot T)$, where m and n are the state and control dimensions and T is the number of steps propogated through.\n", + "While FoPG's have been shown to be very sample efficient, especially as the [dimension of the state space increases](https://arxiv.org/abs/2204.07137), one fundamental shortcoming is that due to the lower gradient variance, FoPG's also have less exploration power than ZoPG's and benefit from the practioner being more explicit in the problem formulation.\n", "\n", - "Due to the lower gradient variance, FoPG's also have less exploration power than ZoPG's and benefit from the practioner being more explicit in the problem formulation. " + "Additionally, discontinuous reward formulations are ubiquitious in RL, for instance, a large penalty when the robot falls. It can be significantly more [challenging](https://arxiv.org/abs/2403.14864) to design robust policies with FoPG's, since they cannot backprop through such penalties.\n", + "\n", + "Last, despite the sample efficiency, FoPG methods can still struggle with wall-clock time. Because the gradients have low variance, they do not benefit significantly from massive parallelization of data collection - unlike [RL](https://arxiv.org/abs/2109.11978). Additionally, the policy gradient is typically calculated via autodifferentiation. This can be 3-5x slower than unrolling the simulation forward, and memory intensive, with memory requirements scaling with $O((m+n) \\cdot m \\cdot T)$, where m and n are the state and control dimensions, $(m+n) \\cdot m$ is the jacobian dimension, and T is the number of steps propogated through." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "In this tutorial, we hope to convey *when* to use FoPG's, and *how* to use them through three case studies:\n", + "In this tutorial, we demonstrate two ways to use FoPG's, using a simple variation of Brax's APG [algorithm](https://github.com/Andrew-Luo1/brax_new_apg/tree/main/brax/training/agents/apg). This algorithm essentially performs live stochastic gradient descent on the policy, unrolling it for a short window, doing a gradient update, then continuing where it left off." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup: Imports and installations\n", "\n", - "\n", - "| | | Case Study |\n", - "| --- | ----| --------------------------------- |\n", - "| 1 | | *Imitating Kinematics* |\n", - "| | a | 1 Hz Trot |\n", - "| | b | 3 Hz Trot |\n", - "| | c | 3 Hz Trot; PPO |\n", - "| 2 | | *Quadruped Locomotion Design* |\n", - "| | a | 0.75 m/s trot with long strides |\n", - "| | b | 1.5 m/s trot with short strides |\n", - "| | c | 0.75 m/s trot without reference |\n", - "\n", - "(TODO)\n", - "Study 1a demonstrates the sample efficiency and degree of refinement possible from a FoPG algorithm. Studies 1a, 2a and 2b show that FoPG algorithms excel when the reward is specified clearly. Study 3 shows that like for classical methods such as MPC and trajectory optimization, the learned policy benefits greatly from a good \"initial guess\"." + "The examples below require a modified version of Brax's Analytical Policy Gradient (APG) algorithm." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": {}, "outputs": [], - "source": [] + "source": [ + "!pip install git+https://github.com/Andrew-Luo1/brax_new_apg.git#egg=brax" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"0.8\" # 0.9 causes too much lag. \n", + "from datetime import datetime\n", + "import functools\n", + "\n", + "# Math\n", + "import jax.numpy as jp\n", + "import numpy as np\n", + "import jax\n", + "from jax import config # Analytical gradients work much better with double precision.\n", + "config.update(\"jax_debug_nans\", True)\n", + "jax.config.update(\"jax_enable_x64\", True)\n", + "jax.config.update('jax_default_matmul_precision', jax.lax.Precision.HIGH)\n", + "import matplotlib.pyplot as plt\n", + "\n", + "# Sim\n", + "import mujoco\n", + "import mujoco.mjx as mjx\n", + "\n", + "# Brax\n", + "from brax import envs\n", + "from brax import envs\n", + "from brax import math\n", + "from brax.base import Motion, Transform\n", + "from brax.io import mjcf\n", + "from brax.envs.base import PipelineEnv, State\n", + "from brax.mjx.pipeline import _reformat_contact\n", + "from brax.training.acme import running_statistics\n", + "from brax.io import model\n", + "\n", + "# Algorithms\n", + "from brax.training.agents.apg import train as apg\n", + "from brax.training.agents.apg import networks as apg_networks\n", + "from brax.training.agents.ppo import train as ppo\n", + "\n", + "# Misc\n", + "import mediapy as media\n", + "from etils import epath\n", + "from ml_collections import config_dict\n", + "from typing import Any, Dict\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Anymal C" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "xml_path = os.getcwd() + \"/mujoco/mjx/test_data/anymal_c/assets/anymal_c.xml\"\n", + "\n", + "mj_model = mujoco.MjModel.from_xml_path(xml_path)\n", + "\n", + "if 'renderer' not in dir():\n", + " renderer = mujoco.Renderer(mj_model)\n", + "\n", + "init_q = mj_model.keyframe('standing').qpos\n", + "\n", + "mj_data = mujoco.MjData(mj_model)\n", + "mj_data.qpos = init_q\n", + "mujoco.mj_forward(mj_model, mj_data)\n", + "\n", + "renderer.update_scene(mj_data)\n", + "media.show_image(renderer.render())" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "# Rendering Rollouts\n", + "def render_rollout(reset_fn, step_fn, \n", + " inference_fn, env, \n", + " n_steps = 250, camera=None,\n", + " seed=0):\n", + " rng = jax.random.key(seed)\n", + " render_every = 2\n", + " state = reset_fn(rng)\n", + " rollout = [state.pipeline_state]\n", + "\n", + " for i in range(n_steps):\n", + " act_rng, rng = jax.random.split(rng)\n", + " ctrl, _ = inference_fn(state.obs, act_rng)\n", + " state = step_fn(state, ctrl)\n", + " if i % render_every == 0:\n", + " rollout.append(state.pipeline_state)\n", + "\n", + " media.show_video(env.render(rollout, camera=camera), fps=1.0 / (env.dt*render_every))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Study 1: Imitating Kinematics\n", + "\n", + "FoPG's have been demonstrated to work well in [imitation learning](https://openreview.net/forum?id=06mk-epSwZ), especially when the agent state is reset to track the reference state when it gets too far. In this section, we learn to trot in place. Due to the limited gain on the PD controllers, trotting quickly is far from trivial!\n", + "\n", + "The RL environment has three rewards:\n", + "- min_reference_tracking penalizes error from the reference motion, in minimal coordinates. This makes the policy's output more precise.\n", + "- reference_tracking penalizes error in maximal coordinates and increases training stability.\n", + "- feet_height nudges the balance of which body positions and velocities to track, by placing additional incentive on the *position* of the feet." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Designing Reference Kinematics" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "def cos_wave(t, step_period, scale):\n", + " _cos_wave = -jp.cos(((2*jp.pi)/step_period)*t)\n", + " return _cos_wave * (scale/2) + (scale/2)\n", + "\n", + "def dcos_wave(t, step_period, scale):\n", + " \"\"\" \n", + " Derivative of the cos wave, for reference velocity\n", + " \"\"\"\n", + " return ((scale*jp.pi) / step_period) * jp.sin(((2*jp.pi)/step_period)*t)\n", + "\n", + "def make_kinematic_ref(sinusoid, step_k, scale=0.3, dt=1/50):\n", + " \"\"\" \n", + " Makes trotting kinematics for the 12 leg joints.\n", + " \"\"\"\n", + " \n", + " _steps = jp.arange(step_k)\n", + " step_period = step_k * dt\n", + " t = _steps * dt\n", + " \n", + " wave = sinusoid(t, step_period, scale)\n", + " f_leg_cmd_bloc = jp.concatenate(\n", + " [jp.zeros((step_k, 1)),\n", + " wave.reshape(step_k, 1),\n", + " -2*wave.reshape(step_k, 1)],\n", + " axis=1\n", + " )\n", + " h_leg_cmd_bloc = -1 * f_leg_cmd_bloc\n", + "\n", + " bloc1 = jp.concatenate([\n", + " jp.zeros((step_k, 3)),\n", + " f_leg_cmd_bloc,\n", + " h_leg_cmd_bloc,\n", + " jp.zeros((step_k, 3))],\n", + " axis=1\n", + " )\n", + "\n", + " bloc2 = jp.concatenate([\n", + " f_leg_cmd_bloc,\n", + " jp.zeros((step_k, 3)),\n", + " jp.zeros((step_k, 3)),\n", + " h_leg_cmd_bloc],\n", + " axis=1\n", + " )\n", + "\n", + " cycle = jp.concatenate([bloc1, bloc2], axis=0)\n", + " return cycle\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "poses = make_kinematic_ref(cos_wave, step_k=25)\n", + "\n", + "frames = []\n", + "init_q = mj_model.keyframe('standing').qpos\n", + "mj_data.qpos = init_q\n", + "default_ap = init_q[7:]\n", + "\n", + "for i in range(len(poses)):\n", + " mj_data.qpos[7:] = poses[i] + default_ap\n", + " mujoco.mj_forward(mj_model, mj_data)\n", + " renderer.update_scene(mj_data)\n", + " frames.append(renderer.render())\n", + "\n", + "media.show_video(frames, fps=50, codec='gif')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### The RL Environment" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "def get_config():\n", + " def get_default_rewards_config():\n", + " default_config = config_dict.ConfigDict(\n", + " dict(\n", + " scales=config_dict.ConfigDict(\n", + " dict(\n", + " min_reference_tracking = -2.5 * 3e-3, # to equalize the magnitude\n", + " reference_tracking = -1.0,\n", + " feet_height = -1\n", + " )\n", + " )\n", + " )\n", + " )\n", + " return default_config\n", + "\n", + " default_config = config_dict.ConfigDict(\n", + " dict(rewards=get_default_rewards_config(),))\n", + "\n", + " return default_config\n", + "\n", + "# Math functions from (https://github.com/jiawei-ren/diffmimic)\n", + "def quaternion_to_matrix(quaternions):\n", + " r, i, j, k = quaternions[..., 0], quaternions[..., 1], quaternions[..., 2], quaternions[..., 3]\n", + " two_s = 2.0 / (quaternions * quaternions).sum(-1)\n", + "\n", + " o = jp.stack(\n", + " (\n", + " 1 - two_s * (j * j + k * k),\n", + " two_s * (i * j - k * r),\n", + " two_s * (i * k + j * r),\n", + " two_s * (i * j + k * r),\n", + " 1 - two_s * (i * i + k * k),\n", + " two_s * (j * k - i * r),\n", + " two_s * (i * k - j * r),\n", + " two_s * (j * k + i * r),\n", + " 1 - two_s * (i * i + j * j),\n", + " ),\n", + " -1,\n", + " )\n", + " return o.reshape(quaternions.shape[:-1] + (3, 3))\n", + "\n", + "def matrix_to_rotation_6d(matrix):\n", + " batch_dim = matrix.shape[:-2]\n", + " return matrix[..., :2, :].reshape(batch_dim + (6,))\n", + "\n", + "def quaternion_to_rotation_6d(quaternion):\n", + " return matrix_to_rotation_6d(quaternion_to_matrix(quaternion))\n", + "\n", + "class TrotAnymal(PipelineEnv):\n", + "\n", + " def __init__(\n", + " self,\n", + " termination_height: float=0.25,\n", + " **kwargs,\n", + " ):\n", + " step_k = kwargs.pop('step_k', 25)\n", + "\n", + " physics_steps_per_control_step = 10\n", + " kwargs['n_frames'] = kwargs.get(\n", + " 'n_frames', physics_steps_per_control_step)\n", + "\n", + " mj_model = mujoco.MjModel.from_xml_path(xml_path)\n", + " mj_model.opt.timestep = 0.002\n", + " kp = 230\n", + " mj_model.actuator_gainprm[:, 0] = kp\n", + " mj_model.actuator_biasprm[:, 1] = -kp\n", + "\n", + " sys = mjcf.load_model(mj_model)\n", + "\n", + " super().__init__(sys=sys, **kwargs) \n", + " \n", + " self.termination_height = termination_height\n", + " \n", + " self._init_q = mj_model.keyframe('standing').qpos\n", + " \n", + " self.err_threshold = 0.4 # diffmimic; value from paper.\n", + " \n", + " self._default_ap_pose = mj_model.keyframe('standing').qpos[7:]\n", + " self.reward_config = get_config()\n", + "\n", + " self.action_loc = self._default_ap_pose\n", + " self.action_scale = jp.array([0.2, 0.8, 0.8] * 4)\n", + " \n", + " self.feet_inds = jp.array([21,28,35,42]) # LF, RF, LH, RH\n", + "\n", + " #### Imitation reference\n", + " kinematic_ref_qpos = make_kinematic_ref(\n", + " cos_wave, step_k, scale=0.3, dt=self.dt)\n", + " kinematic_ref_qvel = make_kinematic_ref(\n", + " dcos_wave, step_k, scale=0.3, dt=self.dt)\n", + " \n", + " self.l_cycle = jp.array(kinematic_ref_qpos.shape[0])\n", + " \n", + " # Expand to entire state space.\n", + "\n", + " kinematic_ref_qpos += self._default_ap_pose\n", + " ref_qs = np.tile(self._init_q.reshape(1, 19), (self.l_cycle, 1))\n", + " ref_qs[:, 7:] = kinematic_ref_qpos\n", + " self.kinematic_ref_qpos = jp.array(ref_qs)\n", + " \n", + " ref_qvels = np.zeros((self.l_cycle, 18))\n", + " ref_qvels[:, 6:] = kinematic_ref_qvel\n", + " self.kinematic_ref_qvel = jp.array(ref_qvels)\n", + "\n", + " # Can decrease jit time and training wall-clock time significantly.\n", + " self.pipeline_step = jax.checkpoint(self.pipeline_step, \n", + " policy=jax.checkpoint_policies.dots_with_no_batch_dims_saveable)\n", + " \n", + " def reset(self, rng: jax.Array) -> State:\n", + " # Deterministic init\n", + "\n", + " qpos = jp.array(self._init_q)\n", + " qvel = jp.zeros(18)\n", + " \n", + " data = self.pipeline_init(qpos, qvel)\n", + "\n", + " # Position onto ground\n", + " pen = jp.min(data.contact.dist)\n", + " qpos = qpos.at[2].set(qpos[2] - pen)\n", + " data = self.pipeline_init(qpos, qvel)\n", + "\n", + " state_info = {\n", + " 'rng': rng,\n", + " 'steps': 0.0,\n", + " 'reward_tuple': {\n", + " 'reference_tracking': 0.0,\n", + " 'min_reference_tracking': 0.0,\n", + " 'feet_height': 0.0\n", + " },\n", + " 'last_action': jp.zeros(12), # from MJX tutorial.\n", + " 'kinematic_ref': jp.zeros(19),\n", + " }\n", + "\n", + " x, xd = data.x, data.xd\n", + " obs = self._get_obs(data.qpos, x, xd, state_info)\n", + " reward, done = jp.zeros(2)\n", + " metrics = {}\n", + " for k in state_info['reward_tuple']:\n", + " metrics[k] = state_info['reward_tuple'][k]\n", + " state = State(data, obs, reward, done, metrics, state_info)\n", + " return jax.lax.stop_gradient(state)\n", + " \n", + " def step(self, state: State, action: jax.Array) -> State:\n", + " action = jp.clip(action, -1, 1) # Raw action\n", + "\n", + " action = self.action_loc + (action * self.action_scale)\n", + "\n", + " data = self.pipeline_step(state.pipeline_state, action)\n", + " \n", + " ref_qpos = self.kinematic_ref_qpos[jp.array(state.info['steps']%self.l_cycle, int)]\n", + " ref_qvel = self.kinematic_ref_qvel[jp.array(state.info['steps']%self.l_cycle, int)]\n", + " \n", + " # Calculate maximal coordinates\n", + " ref_data = data.replace(qpos=ref_qpos, qvel=ref_qvel)\n", + " ref_data = mjx.forward(self.sys, ref_data)\n", + " ref_x, ref_xd = ref_data.x, ref_data.xd\n", + "\n", + " state.info['kinematic_ref'] = ref_qpos\n", + "\n", + " # observation data\n", + " x, xd = data.x, data.xd\n", + " obs = self._get_obs(data.qpos, x, xd, state.info)\n", + "\n", + " # Terminate if flipped over or fallen down.\n", + " done = 0.0\n", + " done = jp.where(x.pos[0, 2] < self.termination_height, 1.0, done)\n", + " up = jp.array([0.0, 0.0, 1.0])\n", + " done = jp.where(jp.dot(math.rotate(up, x.rot[0]), up) < 0, 1.0, done)\n", + "\n", + " # reward\n", + " reward_tuple = {\n", + " 'reference_tracking': (\n", + " self._reward_reference_tracking(x, xd, ref_x, ref_xd)\n", + " * self.reward_config.rewards.scales.reference_tracking\n", + " ),\n", + " 'min_reference_tracking': (\n", + " self._reward_min_reference_tracking(ref_qpos, ref_qvel, state)\n", + " * self.reward_config.rewards.scales.min_reference_tracking\n", + " ),\n", + " 'feet_height': (\n", + " self._reward_feet_height(data.geom_xpos[self.feet_inds][:, 2]\n", + " ,ref_data.geom_xpos[self.feet_inds][:, 2])\n", + " * self.reward_config.rewards.scales.feet_height\n", + " )\n", + " }\n", + " \n", + " reward = sum(reward_tuple.values())\n", + "\n", + " # state management\n", + " state.info['reward_tuple'] = reward_tuple\n", + " state.info['last_action'] = action # used for observation. \n", + "\n", + " for k in state.info['reward_tuple'].keys():\n", + " state.metrics[k] = state.info['reward_tuple'][k]\n", + "\n", + " state = state.replace(\n", + " pipeline_state=data, obs=obs, reward=reward,\n", + " done=done)\n", + " \n", + " #### Demonstration Replay\n", + " error = (((x.pos - ref_x.pos) ** 2).sum(-1)**0.5).mean()\n", + " replay = jp.where(error > self.err_threshold, 1.0, 0.0)\n", + "\n", + " replay = jp.array(replay, dtype=int) # keeps output types same as input. \n", + " ref_data = self.mjx_to_brax(ref_data)\n", + "\n", + " data = jax.tree_util.tree_map(lambda x, y: \n", + " jp.array((1-replay)*x + replay*y, x.dtype), data, ref_data)\n", + " \n", + " x, xd = data.x, data.xd # Data may or may not have changed.\n", + " obs = self._get_obs(data.qpos, x, xd, state.info)\n", + " \n", + " return state.replace(pipeline_state=data, obs=obs)\n", + " \n", + " def _get_obs(self, qpos: jax.Array, x: Transform, xd: Motion,\n", + " state_info: Dict[str, Any]) -> jax.Array:\n", + "\n", + " inv_base_orientation = math.quat_inv(x.rot[0])\n", + " local_rpyrate = math.rotate(xd.ang[0], inv_base_orientation)\n", + "\n", + " obs_list = []\n", + " # yaw rate\n", + " obs_list.append(jp.array([local_rpyrate[2]]) * 0.25)\n", + " # projected gravity\n", + " obs_list.append(\n", + " math.rotate(jp.array([0.0, 0.0, -1.0]), inv_base_orientation))\n", + " # motor angles\n", + " angles = qpos[7:19]\n", + " obs_list.append(angles - self._default_ap_pose)\n", + " # last action\n", + " obs_list.append(state_info['last_action'])\n", + " # kinematic reference\n", + " kin_ref = self.kinematic_ref_qpos[jp.array(state_info['steps']%self.l_cycle, int)]\n", + " obs_list.append(kin_ref) # Gait schedule, per actuator.\n", + "\n", + " obs = jp.clip(jp.concatenate(obs_list), -100.0, 100.0)\n", + "\n", + " return obs\n", + " \n", + " def mjx_to_brax(self, data):\n", + " q, qd = data.qpos, data.qvel\n", + " x = Transform(pos=data.xpos[1:], rot=data.xquat[1:])\n", + " cvel = Motion(vel=data.cvel[1:, 3:], ang=data.cvel[1:, :3])\n", + " offset = data.xpos[1:, :] - data.subtree_com[self.sys.body_rootid[1:]]\n", + " offset = Transform.create(pos=offset)\n", + " xd = offset.vmap().do(cvel)\n", + " data = _reformat_contact(self.sys, data)\n", + " return data.replace(q=q, qd=qd, x=x, xd=xd)\n", + "\n", + "\n", + " # ------------ reward functions----------------\n", + " def _reward_reference_tracking(self, x, xd, ref_x, ref_xd):\n", + " \"\"\"\n", + " Rewards based on inertial-frame body positions.\n", + " Notably, we use a high-dimension representation of orientation.\n", + " \"\"\"\n", + "\n", + " f = lambda x, y: ((x - y) ** 2).sum(-1).mean()\n", + "\n", + " _mse_pos = f(x.pos, ref_x.pos)\n", + " _mse_rot = f(quaternion_to_rotation_6d(x.rot),\n", + " quaternion_to_rotation_6d(ref_x.rot))\n", + " _mse_vel = f(xd.vel, ref_xd.vel)\n", + " _mse_ang = f(xd.ang, ref_xd.ang)\n", + "\n", + " # Tuned to be about the same size.\n", + " return _mse_pos \\\n", + " + 0.1 * _mse_rot \\\n", + " + 0.01 * _mse_vel \\\n", + " + 0.001 * _mse_ang\n", + "\n", + " def _reward_min_reference_tracking(self, ref_qpos, ref_qvel, state):\n", + " \"\"\" \n", + " Using minimal coordinates. Improves accuracy of joint angle tracking.\n", + " \"\"\"\n", + " pos = jp.concatenate([\n", + " state.pipeline_state.qpos[0:3],\n", + " state.pipeline_state.qpos[4:]])\n", + " pos_targ = jp.concatenate([\n", + " ref_qpos[0:3],\n", + " ref_qpos[4:]])\n", + " pos_err = jp.linalg.norm(pos_targ - pos)\n", + " vel_err = jp.linalg.norm(state.pipeline_state.qvel- ref_qvel)\n", + "\n", + " return pos_err + vel_err\n", + "\n", + " def _reward_feet_height(self, feet_pos, feet_pos_ref):\n", + " return jp.sum(jp.abs(feet_pos - feet_pos_ref)) # try to drive it to 0 with l1 norm\n", + "\n", + "envs.register_environment('trotting_anymal', TrotAnymal)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Imitation Learning via FoPG's\n", + "Takes 15 minutes on a NVIDIA 3060 TI GPU" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "make_networks_factory = functools.partial(\n", + " apg_networks.make_apg_networks,\n", + " hidden_layer_sizes=(256, 128)\n", + ")\n", + "\n", + "epochs = 500\n", + "\n", + "train_fn = functools.partial(apg.train,\n", + " episode_length=240,\n", + " policy_updates=epochs,\n", + " horizon_length=32,\n", + " num_envs=64,\n", + " learning_rate=1e-4,\n", + " num_eval_envs=64,\n", + " num_evals=10 + 1,\n", + " normalize_observations=True,\n", + " network_factory=make_networks_factory)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "x_data = []\n", + "y_data = []\n", + "ydataerr = []\n", + "times = [datetime.now()]\n", + "\n", + "def progress(it, metrics):\n", + " times.append(datetime.now())\n", + " x_data.append(it)\n", + " y_data.append(metrics['eval/episode_reward'])\n", + " ydataerr.append(metrics['eval/episode_reward_std'])\n", + "\n", + "# Each foot contacts the ground twice/sec.\n", + "env = envs.get_environment(\"trotting_anymal\", step_k = 13)\n", + "eval_env = envs.get_environment(\"trotting_anymal\", step_k = 13)\n", + "\n", + "make_inference_fn, params, _= train_fn(environment=env,\n", + " progress_fn=progress,\n", + " eval_env=eval_env)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAisAAAGdCAYAAADT1TPdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA+VUlEQVR4nO3deXhU9aH/8c9kmywkk8BAFrKRuGBkT1gCVEQtYKleS9VLVQSlUK6kLmgVilcWBbRAf/dWaxEX1KrVa12BeosKV4tsAQIiKMoSCAkJSyATEjJZ5vz+CIzEhCWYyZmZvF/PM0/nnDkz+WRsmM/znfP9HothGIYAAAC8VIDZAQAAAM6FsgIAALwaZQUAAHg1ygoAAPBqlBUAAODVKCsAAMCrUVYAAIBXo6wAAACvFmR2gB/L5XKpqKhIkZGRslgsZscBAAAXwDAMlZeXKyEhQQEB5x478fmyUlRUpKSkJLNjAACAi1BQUKDExMRzHuPzZSUyMlJS/S8bFRVlchoAAHAhHA6HkpKS3J/j5+LzZeX0Vz9RUVGUFQAAfMyFnMLBCbYAAMCrUVYAAIBXo6wAAACvRlkBAABejbICAAC8GmUFAAB4NcoKAADwapQVAADg1SgrAADAq1FWAACAV6OsAAAAr0ZZAQAAXo2yAgAAvBplBQAAD6usrlXq1OVKnbpcldW1ZsfxOZQVAADg1SgrAADAq1FWAACAV6OsAAAAr0ZZAQAAXo2yAgAAvBplBQAAeDXKCgAA8GqUFQAA4NUoKwAAoEnesvIuZQUAAHg1ygoAAPBqlBUAAODVKCsAAMCreays5Ofna/z48erSpYvCwsKUnp6uGTNmqLq6usFxFoul0W3RokWeigUAAHxMkKde+JtvvpHL5dJzzz2nSy65RF999ZUmTJigiooKLViwoMGxS5Ys0YgRI9zbNpvNU7EAAICP8VhZGTFiRIMCkpaWpp07d+ovf/lLo7ISHR2tuLg4T0UBAAA+rFXPWSkrK1P79u0b7c/JyZHdblffvn21aNEiuVyus76G0+mUw+FocAMAAP7LYyMrP7R79249/fTTWrhwYYP9jz/+uK699lqFhYXp008/1YMPPqgjR47o0UcfbfJ15s2bp1mzZrVGZAAA2rST1XVmR5AkWQzDMJrzhJkzZ563LOTm5iorK8u9XVRUpCFDhmjIkCF64YUXzvnchQsXavbs2SorK2vycafTKafT6d52OBxKSkpSWVmZoqKimvGbAADQOiqra5Xx2D8lSTtmD1d4SKuNFTRLTZ1LXx44ri92HdUXu45o8/5jqqmrrwktndvhcMhms13Q53ezf2pOTo5Gjx59zmNSU1Pd94uKijR06FBlZ2dr8eLF5339AQMGyOFwqKSkRLGxsY0et1qtslqtzY0NAAB+wOUy9E1xudbsPqI1u49q/Z6jqvCS0ZQzNbus2O122e32Czq2sLBQQ4cOVWZmppYsWaKAgPOfIpOXl6fQ0FBFR0c3NxoAoA3wlVEKb2QYhvaXVtaPnOw+onW7j+poRcMlRWLCg5Wd3kED0+3qkxKtn/33apPSfs9j/4WLiop09dVXKzk5WQsWLNDhw4fdj52e+bN06VIVFxcrOztbYWFhWrVqlaZPn66JEycyegIAQAs4VF6lNbuOas3uI/pi11EVHj/Z4PGw4ED169Jegy6pLygZ8VEKCLBIkqkXLzyTx8rKihUrtGvXLu3atUuJiYkNHjt9mkxwcLCeffZZTZkyRS6XS2lpaZo9e7YmT57sqVgAAPi1spM1Wr/nqNbsrj/v5LtDJxo8HhxoUe+kGA28pIMGXWJXz8RohQR594L2Hisr48aN07hx4855zA/XYgEAAM1TVVOnjfnH6kdOdh/VtgPH5Tpj6ozFImXER2nQJXYNTO+gfl3a+9xXZ76VFgCANq62zqUvC8u0Zlf91zqb9h9TdW3D9cnS7BH1Iyfpdg1I66CYiBCT0rYMygoAAF7MMAztLCl3n3eyfk+pyp0NzyWJjbJqULpdAy+xa9AlHRRvCzMprWdQVgAA8DIFpZX6Ylf91zprdx/RkRMNZ+zYwoKVndah/qTYS+xKs0fIYrGYlNbzKCsAAJjscLlTa/ccrf9qZ/cRFZQ2nLETGhygvqntNegSuwal25WREKXAAP8tJz9EWQEAoJWVV9Vo/Z5SfbH7iNbsOqqdJeUNHg8KsKhXUnT91zrpHdQrOVrWoECT0pqPsgIAQAurrnXpeGW1jlXWqLSiWiWO70dKfrV4nb4qcqjO1fBqN1fER2lQev104r5d2qudlY/o03gnAAA4h6qaOpVWVOtYZbWOnyofxyurVVpRc2pftUora07tqz/mhPPsi6ltPVB/7bvUDuGnRk7syk7voPZeOGMnPCRI+U+ONDsGZQUA0DYYhqGK6jodO1U8jlXWnOV+tY5VnCofldWqqnGd/8WbEGCRosNDFB0erOiwYG3ef1yS9MRN3TS0ayd1jvavGTueRFkBAPisfUcrdLLm1Fcup0Y6jp0a9Th+RvE4vf/0FYSbKyjAopiIEMWEBysmPKT+FnHm/frHosND1P7U/ajQ4AbL1p++ntGoPp19blE2s/FuAQB8hstl6P92HnJvX38RF9mzBgWofUSIosNPlY9T5aJ9+Kl9Z5SQ+uOC1c4a5NdTg70dZQUA4PWqa136YEuhFn++p8G1bsJDAhuUivpRje/vR4eHnCoh9aWkfXiIwkLa7qwaX0VZAQB4rfKqGv1tw369tDpfxY4qSVKENVAVzjpJ0sZHr+MrlTaA/8IAAK9zyFGll77I1+vr9rmXlu8UadX4wV10U+8E9Z+70uSEaE2UFQCA19h16ISe/3yP3ssrVHVd/Syc9I4R+s2QdP1brwRZgwJVWX32acHwT5QVAIDpNu0r1aLP9ujjHSXufVkpMZo0JF3XdO3knlWDtomyAgBt2JlTanfMHt6q53+4XIY+/eaQnvtstzbuO+bePywjVr8ZkqbMlPatlgXejbICAGhVzto6fZBXpOc+363dhyskSSGBARrVp7N+/ZM0XdKpnckJ4W0oKwDQQswcpfAFjqoavbF+v15avVeHyp2SpEhrkG4fkKK7BqUqNirU5ITwVvwlAQA8qrisSku+2KvX1+93XzMnLipUdw9O1a/6JSsyNNjkhPB2lBUAgEd8V1KuxZ/v0ftbCt3L3F/aqZ1+MyRdN/ZMUEhQgMkJ4SsoKwCAFmMYhjbuO6bnPtutT77+fln8fqnt9ZshaRp6OTN70HyUFQDAj+ZyGfr46xI999lu99WFLZb6mT0Tr0pXZkqMuQHh0ygrAICLVlVTp/fzCrX4X3u054yZPb/MrJ/Zk96RmT348SgrALwSM2u8W9nJGr2+fp+WfJGvw6dm9kSFBumOASkaNyhVnSKZ2YOWw18/AOCCHSw7qZdW79Ub6/erorr+YoLxtlCNH9xFo/slq53V8x8r4SFByn9ypMd/DrwHZQUAcF47i+tn9nywpVC1rvqZPZfFttNvrkrXDczsgYdRVgAATTIMQxv2luq5z/do5Tffz+zp36W9Jg1J19WXd5TFwsweeB5lBQDQQJ3L0Mc7irXosz3aUnBcUv3MnhFXxmniVWnqnczMHrQuygoAQJLkrKnT+3lFev5fe7T3yKmZPUEBujkzURN+kqYu9giTE6KtoqwAQBtT5zJ0wlmrE85aHSmvcu+/7v99rqMnqiXVz+y5MztVYwemqmOk1ayogCQPl5Ubb7xRW7Zs0aFDhxQTE6PrrrtOTz31lBISEtzH7N+/X5MnT9bKlSsVFham2267TQsWLFBISIgnowGAz3HW1ulEVX3JKD/1v+5t9/0anag6c/v7407vO1lT1+TrHz1RrQRbqMb/JE2j+yYpohVm9gAXwqP/Txw6dKh+//vfKz4+XoWFhXrooYd08803a82aNZKkuro6jRw5Uh07dtTq1at19OhRjR07VoZh6Omnn/ZkNABoFYZhqLK6romCUXOOwnFGuThVPk44a93X12kpIUEBamcNUmlF/WjKk7/srl/2SVRwIDN74F08WlYeeOAB9/2UlBRNnTpVN910k2pqahQcHKwVK1Zox44dKigocI+2LFy4UOPGjdOcOXMUFRXlyXgA0CKOnHBq24EybdpX6t53zYLPVOGs1YnqWhkt2zEUERKodqFBamcNUrvQYEVaT9+v/9/I0Ibb7c54PNIarHahQYqwBsoaFNhg8b0beyZQVOCVWm2Mr7S0VK+//roGDhyo4OD6y4GvXbtW3bp1a/C10PDhw+V0OrVp0yYNHTq00es4nU45nU73tsPh8Hx4ADjlWEW1thWWaVthmb48cFzbDpSpqKyq0XHFjob7AgMs35eIs5WLU0Ui8syiccZ2hDVIESFBCuRCgGhjPF5WHnnkET3zzDOqrKzUgAEDtGzZMvdjxcXFio2NbXB8TEyMQkJCVFxc3OTrzZs3T7NmzfJoZgCQJEdVjb46UKYvC8u07UCZviw8roLSk42Os1ik9I7tlBEfpQ+3FkmS3p40QPZ2oe4yYg0KYE0S4CI1u6zMnDnzvGUhNzdXWVlZkqTf/e53Gj9+vPbt26dZs2bpzjvv1LJly9x/tE398RqGcdY/6mnTpmnKlCnubYfDoaSkpOb+GgDQQIWzVtuLHPWjJafKyZ5T03d/qIs9Qt0729Qj0abunW26srNN7axBqqyudZeVKxNsXM8IaCHN/kvKycnR6NGjz3lMamqq+77dbpfdbtdll12mK664QklJSVq3bp2ys7MVFxen9evXN3jusWPHVFNT02jE5TSr1SqrlWl0AC7eyeo67Tjo0LYDx92jJrsOn2jy3JKk9mHq0Tla3RNt6nGqmNjCgls/NNCGNbusnC4fF8M49S/B6XNOsrOzNWfOHB08eFDx8fGSpBUrVshqtSozM/OifgYAnMlZW6dvDpafKiXH9eWBMn136ITqXI2bSYIttL6UJEare+f6UZOYCJZRAMzmsTHKDRs2aMOGDRo8eLBiYmK0Z88ePfbYY0pPT1d2drYkadiwYcrIyNCYMWM0f/58lZaW6qGHHtKECROYCQSg2aprXfq2pPzUya9l2lZ4XDuLy5uc8tsx0qqeiTZ17xytHok2detsY/EzwEt5rKyEhYXp3Xff1YwZM1RRUaH4+HiNGDFCb775pvtrnMDAQC1fvlz33HOPBg0a1GBROAA4l9o6l3YdPlFfSk6dBPv1QYeqa12Njm0fEaIep77G6Z5YX05io0JNSI22KjwkSPlPjjQ7hs/yWFnp3r27Vq5ced7jkpOTG8wQAoAfqnMZ2nWoXF8eOD1iUqbtRWWqqmlcTGxhwe4TX3sk1peTBFsoM3EAH8ap6gC80pkjJP3nfqrK6sZLxEdag9TNXUps6tE5WkntwygmgJ+hrADwOp9+XaKZS7e7tyur6xQeEqhuCadKyamRk9QOEQpggTTA71FWAHiNvUcqNHvpdq3aebjB/g9zBunKBBsrtwJtFGUFgOkqnLV6euUuvbh6j2rqDAUHWnRndopeXJ0vSbqkUzuKCtCGUVYAmMYwDH24tUhz//G1Shz16y8NuayjHrshQ/G2UHdZAdC2UVYAmGJ7UZlmfrhdufnHJEnJ7cP12M8zdO0VnWSxWFRZXWtyQgDegrICoFUdq6jWghU79bcN++UypLDgQOVcc4nGD+6i0OBAs+O1Oaz/AV9AWQHQKupcht5Yv08LVnyrspM1kqQbeiZo2vVdlRAdZnI6AN6MsgK0AZXVtcp47J+SpB2zh7f61YA37C3VjA+36+uDDklS17hIzbzxSg1I69CqOTyNUQrAMygrADymuKxKc//xtT7cWiSpfnXZB4ddptv6JSsoMMDkdAB8BWUFQItz1tbphX/t1Z9X7VJldZ0sFulX/ZL10LDL1Z6rGANoJsoKgBb16dclmr1sh/YdrZQkZabEaNaNV6pbZ5vJyQD4KsoKgBbxw9VnO0VaNe1nXXVTr85cqwfAj0JZAfCjNLX67N2Du+i311yqdlb+iQHw4/EvCYCLYhiGPthSpHkfNV59Nr1jO5PTAfAnlBUAzfZVYf3qsxv3Nb36LAC0JMoKgAvG6rMAzEBZAXBeZ1t99vc/66p4G6vPAvAsygqAc1q/56hmLt3h96vPAvBelBUATTpYdlLz/vENq88CMB1lBUADrD4LwNtQVgC4edPqs1wUEMBplBUA2nP4hB5ftoPVZwF4JcoK0IadcNbqGVafBeDl+NcIaINYfRaAL6GsAG3MjoMOPfmPb1h9FoDPoKwAbcyti9ay+iwAn0JZAdqAbYVl7vsug9VnAfgWygrg5z7eUaJ7/5bn3n7lrr4acnknExMBQPNQVgA/9urafM38cLtcxvf7+nZpb14gALgIrJkN+CGXy9Cc5Tv02Af1ReXmzM5mRwKAi+bRsnLjjTcqOTlZoaGhio+P15gxY1RUVNTgGIvF0ui2aNEiT8YC/FpVTZ1y/rZZz/9rryTpoWGXadaNV5qcCgAunke/Bho6dKh+//vfKz4+XoWFhXrooYd08803a82aNQ2OW7JkiUaMGOHettlaf2lvwB+UVlRrwqsbtWnfMQUHWjT/5p66qXdnVVbXmh0NAC6aR8vKAw884L6fkpKiqVOn6qabblJNTY2Cg4Pdj0VHRysuLs6TUQC/l3+kQne9nKu9RyoUFRqk58ZkKTu9g9mxAOBHa7VzVkpLS/X6669r4MCBDYqKJOXk5Mhut6tv375atGiRXC7XWV/H6XTK4XA0uAFt3aZ9xzTqL2u090iFOkeH6Z3/GEhRAeA3PF5WHnnkEUVERKhDhw7av3+/PvjggwaPP/7443r77bf1ySefaPTo0XrwwQc1d+7cs77evHnzZLPZ3LekpCRP/wqAV/vfrw7qtufXqbSiWt06R+m9yQN1aWyk2bEAoMVYDMMwzn/Y92bOnKlZs2ad85jc3FxlZWVJko4cOaLS0lLt27dPs2bNks1m07Jly866rPfChQs1e/ZslZWVNfm40+mU0+l0bzscDiUlJamsrExRUVHN+VUAn/fi6r16YvkOGYZ0TddOevpXvRXRxAUIK6trlfHYPyVJO2YPV3gIqxYAMJfD4ZDNZrugz+9m/4uVk5Oj0aNHn/OY1NRU93273S673a7LLrtMV1xxhZKSkrRu3TplZ2c3+dwBAwbI4XCopKREsbGxjR63Wq2yWq3NjQ34lTqXoceX7dDLa/IlSXcMSNbMG65UUCCrEQDwP80uK6fLx8U4PYhz5sjID+Xl5Sk0NFTR0dEX9TMAf3eyuk73vpmnj3eUSJKmXt9Vv7kqjYsQAvBbHhsL3rBhgzZs2KDBgwcrJiZGe/bs0WOPPab09HT3qMrSpUtVXFys7OxshYWFadWqVZo+fbomTpzI6AnQhCMnnBr/ykZtLTiukMAALby1p27omWB2LADwKI+VlbCwML377ruaMWOGKioqFB8frxEjRujNN990F5Hg4GA9++yzmjJlilwul9LS0jR79mxNnjzZU7EAn7X78AmNW7JBBaUnFR0erOfvzFLfVJbOB+D/PFZWunfvrpUrV57zmBEjRjRYDA5A03LzSzXh1Y06XlmjpPZhevmufkrv2M7sWADQKpgSAHi5pVuL9ODbW1Vd61LPpGi9ODZL9nZ8TQqg7aCsAF7KMAw99/kePfnRN5Kkn2bE6k+jeyssJNDkZADQuigrgBeqrXNpxofb9fr6/ZKkcQNT9Z8/z1BgADN+ALQ9lBWgmTy9wFqFs1a//VueVn5zSBaL9OjIDI0f3KVFfwYA+BLKCuBFDjmqdPcrufqq0CFrUID+e3QvjegWb3YsADAVZQXwEt+VlGvcklwVHj+p9hEhev7OLGWmxJgdCwBMR1kBvMDa3Uc18a8bVV5Vqy72CC0Z11ep9ogWe/3wkCDlPzmyxV4PAFoTZQUw2Xt5B/Tw379UTZ2hzJQYPX9nltpHhJgdCwC8BmUFMIlhGPrzql1asOJbSdLPusfpj7f2UmgwU5MB4EyUFcAENXUu/ef7X+nN3AJJ0sSr0jR1RFcFMDUZABqhrACtrLyqRpPfyNPn3x5WgEWaeeOVujM71exYAOC1KCtAKyouq9JdL+fq64MOhQYH6Olf9dFPM2LNjgUAXo2yArSSb4odumtJrg6WVcneLkQvju2rnknRZscCAK9HWQFawb++O6z/eG2zTjhrld4xQi/f1U9J7cPNjgUAPoGyAnjY/2ws0O/f3aZal6F+Xdpr8ZhMRYczNRkALhRlBfAQwzD0X598p//+9DtJ0o09EzT/lh6yBjE1GQCag7ICU3n6ooBmqa51adq72/TO5gOSpHuuTtdDwy5najIAXAT/+GQAvIijqkaT/rpJa3YfVWCARY//Wzfd1j/Z7FgA4LMoK0ALKjx+Unct2aBvS04oPCRQf769j4Ze3snsWADg0ygrQAv5qrBMd7+cq0PlTnWKtOqlcX3VrbPN7FgA4PMoK0ALWLXzkHJe36yK6jpdFttOS+7qp87RYWbHAgC/QFkBfqQ31u/Xf37wlepchrLTOmjRmEzZwoLNjgUAfoOyAvwI//XJd1r8+R5J0qjenfXkL3soJCjA5FQA4F8oK8CPcLqo3HvtpXrguktlsTA1GQBaGmUFaIaT1XVa/uVB93ZQgEVzf9Fdt/ZNMjEVAPg3ygpwHi6XoXV7jurdvEL971fFOuGsdT+26I4+ui4jzsR0AOD/KCvAWXxXUq538wr1QV6hisqq3PsTY8J04NhJSdLAS+xmxQOANoOyApzhcLlTH24t0nt5B/RVocO9PzI0SD/vEa9RfRKVER+pK2esMDElALQtlBW0eVU1dVqxo0TvbT6gz787ojqXIan+fJSrL++kUX0665qunRQaXH8Bwsrq2nO9HACghVFW0Ca5XIbW7y3Ve3kH9I9tDc9D6ZkUrVG9O+vnPeLVoZ3VxJQAAKmVyorT6VT//v21detW5eXlqVevXu7H9u/fr8mTJ2vlypUKCwvTbbfdpgULFigkJKQ1oqGN2XWoXO9uLtQHW4pUePyke3/n6DCN6tNZN/XurPSO7UxMCAD4oVYpKw8//LASEhK0devWBvvr6uo0cuRIdezYUatXr9bRo0c1duxYGYahp59+ujWioQ04csKppVuL9F5eob48UObeH2kN0sge8fpF787qm9peAQGskQIA3sjjZeWjjz7SihUr9M477+ijjz5q8NiKFSu0Y8cOFRQUKCEhQZK0cOFCjRs3TnPmzFFUVJSn48FPVdXU6ZOvS/Te5kL937eHG5yHMuSyjvpFn8667opY93koAADv5dGyUlJSogkTJuj9999XeHh4o8fXrl2rbt26uYuKJA0fPlxOp1ObNm3S0KFDGz3H6XTK6XS6tx0OR6Nj0Da5XIZy80v17uZC/WPbQZWfcR5Kj0SbRvXurBt6JnAeCgD4GI+VFcMwNG7cOE2aNElZWVnKz89vdExxcbFiY2Mb7IuJiVFISIiKi4ubfN158+Zp1qxZnogMH7X78Am9t7lQ7+UVNjoP5abeCfpF70Rd0onzUADAVzW7rMycOfO8ZSE3N1dr1qyRw+HQtGnTznlsU9dSMQzjrNdYmTZtmqZMmeLedjgcSkpiqfO2prSiWku3FundzQe09YzzUNpZg/Sz7nH6Re9E9e/CeSgA4A+aXVZycnI0evTocx6TmpqqJ554QuvWrZPV2nDIPSsrS7fffrteeeUVxcXFaf369Q0eP3bsmGpqahqNuJxmtVobvSbahqqaOn369SG9l3dA/7fzsGpPnYcSGGDRVZfa9Ys+ifrpFbEKC+E8FADwJ80uK3a7XXb7+ZcY/9Of/qQnnnjCvV1UVKThw4frrbfeUv/+/SVJ2dnZmjNnjg4ePKj4+HhJ9SfdWq1WZWZmNjca/JDLZWjjvmN6L++Aln15UOVV35+H0q1zlEb1TtQNPRPUMZICCwD+ymPnrCQnJzfYbteu/pyB9PR0JSYmSpKGDRumjIwMjRkzRvPnz1dpaakeeughTZgwgZlAbdyewyf0Xl79eSinr8MjSfG2UN3Uu7NG9e6sS2MjTUwIAGgtpq5gGxgYqOXLl+uee+7RoEGDGiwKh7bnWEW1/r7pgN7dXKgtBcfd+yNCAnV993iN6tNZA7p04DwUAGhjWq2spKamyjCMRvuTk5O1bNmy1ooBLzZk/v+5z0MJsEhXXdZRv+jdWcMy4jgPBQDaMK4NBFMt+SLffb/WZejKhCj9ondn3dgrQZ0iQ80LBgDwGpQVmObNDfs1/5873dsfTB6onkkxJiYCAHijALMDoG1aurVI097b1mAfJ8wCAJpCWUGrW/XNIT3w1hYZhnRrVqLZcQAAXo6ygla1bs9RTXptk2pdhv6tV4L+8+cZZkcCAHg5ygpazZcHjuvXr2yUs9ala7t20oJbeiqQacgAgPPgBFu0iu9KyjX2pQ064axVdloH/fn2PgoODFBNncvsaM0WHhKk/CdHmh0DANoMRlbgcfuPVur2F9brWGWNeiba9PzYLIUGs24KAODCUFbgUSWOKt3x4nodKnfq8thIvXxXP7WzMqAHALhwlBV4TGlFte54Yb32l1YquX24/jq+n2IiQsyOBQDwMZQVeER5VY3GLdmg7w6dUFxUqF7/dX91imJFWgBA81FW0OKqauo0/pWN+vJAmdpHhOi1X/dTUvtws2MBAHwUZQUtqrrWpf94bZM27C1VpDVIr97dT5d0YmVaAMDFo6ygxdS5DE35ny1atfOwQoMD9OK4vurW2WZ2LACAj6OsoEUYhqHp723Tsi8PKjjQokV3ZKpfl/ZmxwIA+AHKCn40wzA09x9f683cAgVYpP/69966+vJOZscCAPgJygp+tGdW7tLz/9orSXpyVA+N7BFvciIAgD+hrOBHefmLvVr48beSpP/8eYZu7ZtkciIAgL+hrOCi/X3TAc1cukOSdN+1l2r84C4mJwIA+CPKCi7K/351UA//fask6e5BXXT/dZeanAgA4K8oK2i2f313WPf+bYtchnRLZqIeHXmFLBaL2bEAAH6KsoJm2ZhfqomvblJ1nUs/6x6nJ3/ZQwEBFBUAgOdQVnDBtheV6a6Xc3Wypk5XXdZR/+/feymQogIA8DDKCi7I7sMndOeLG1ReVau+qTF67o5MWYMCzY4FAGgDKCs4r8LjJzXmhfU6WlGtKxOi9OK4vgoLoagAAFoHZcWPVFbXKnXqcqVOXa7K6toWec3D5U7d8cJ6FZVVKb1jhF69u5+iQoNb5LUBALgQQWYHgPcqq6zRmBfXa++RCnWODtNrv+6vDu2sLfozwkOClP/kyBZ9TQCAf2FkBU2qcNZq3Msb9E1xuTpGWvX6r/sr3hZmdiwAQBtEWUEjVTV1mvjXjcrbf1y2sGD9dXw/pdojzI4FAGijKCtooLbOpd/+LU9f7DqqiJBAvXJ3P3WNizI7FgCgDaOswM3lMvTw37/UxztKFBIUoOfHZqlXUrTZsQAAbVyrlBWn06levXrJYrFoy5YtDR6zWCyNbosWLWqNWDiDYRiauXS73s0rVGCARc/e1kcD0+1mxwIAoHVmAz388MNKSEjQ1q1bm3x8yZIlGjFihHvbZrO1RiycYcGKnXp17T5ZLNIfb+2p6zJizY4EAICkVigrH330kVasWKF33nlHH330UZPHREdHKy4uztNRcBaLPtutP6/aLUl6/N+66d96dTY5EQAA3/Po10AlJSWaMGGC/vrXvyo8PPysx+Xk5Mhut6tv375atGiRXC7XWY91Op1yOBwNbrh4r6/fpyc/+kaS9MiIrrpjQIrJiQAAaMhjZcUwDI0bN06TJk1SVlbWWY97/PHH9fbbb+uTTz7R6NGj9eCDD2ru3LlnPX7evHmy2WzuW1JSkifitwkfbCnUo+9/JUm65+p0/cfV6SYnAgCgsWZ/DTRz5kzNmjXrnMfk5uZqzZo1cjgcmjZt2jmPffTRR933e/XqJUmaPXt2g/1nmjZtmqZMmeLedjgcFJaL8MmOEk35n60yDGnMgBT9bvjlZkcCAKBJzS4rOTk5Gj169DmPSU1N1RNPPKF169bJam24PHtWVpZuv/12vfLKK00+d8CAAXI4HCopKVFsbOOTPK1Wa6PXRPOs2X1E97yxWXUuQ7/o3VmzbrxSFovF7FgAADSp2WXFbrfLbj//lNY//elPeuKJJ9zbRUVFGj58uN566y3179//rM/Ly8tTaGiooqOjmxsNF2BLwXFNeGWjqmtd+mlGrObf3EMBARQVAID38thsoOTk5Abb7dq1kySlp6crMTFRkrR06VIVFxcrOztbYWFhWrVqlaZPn66JEycyeuIBO4vLNfalDaqortOgSzro6V/1VlAg6wICALybqVddDg4O1rPPPqspU6bI5XIpLS1Ns2fP1uTJk82M5Zfyj1TojhfXq+xkjXonR2vxmCyFBgeaHQsAgPNqtbKSmpoqwzAa7BsxYkSDxeDgGQfLTur2F9brcLlTXeMi9fK4foqwmtpTAQC4YHwH4OeOnnDqjhfWq/D4SaV2CNer4/vJFh5sdiwAAC4YZcWPOapqNHbJBu0+XKF4W6he+3V/dYoMNTsWAADNwncBfupkdZ0mvbZJXxU61CEiRK/9ur8SY86+ijAAAN6KkRU/dd+bW5Sbf0yRoUF65e5+Su/YzuxIAABcFMqKn1q964jCggO1ZFxfdevMVawBAL6LsuJHzpxtFRxo0XNjMpWV2t7ERAAA/HiUFT/yz+0l7vsLbumpqy7raGIaAABaBmXFj7y8Jt99/6cZja+rBACAL6Ks+IlN+47pywNlZscAAKDFUVb8xEur95odAQAAj6Cs+IEDxyr10VcHzY4BAIBHUFb8wCtr8uUypAFpzPwBAPgfyoqPO+Gs1ZsbCiRJY7NTzQ0DAIAHUFZ83NsbC1TurFVaxwj95FK72XEAAGhxlBUfVucytOSLfEnSXYO6KCDAYm4gAAA8gLLiwz75ukT7SytlCwvWL/t0NjsOAAAeQVnxYS+emq58W/9khYdwAW0AgH+irPiorwrLtGFvqYICLJxYCwDwa5QVH3V6VGVkj3jF2UJNTgMAgOdQVnxQiaNKS7cWSZLGD+5ichoAADyLsuKDXl2br1qXob6pMeqRGG12HAAAPIqy4mNOVtfp9fX7JTGqAgBoGygrPubdvAM6XlmjpPZh+mlGnNlxAADwOMqKD3G5DPfVlccN7KJAFoEDALQBlBUf8tl3h7X7cIXaWYN0a1ai2XEAAGgVlBUfcnpU5d/7JikyNNjkNAAAtA7Kio/4tqRc//ruiAIs0riBqWbHAQCg1VBWfMTpUZXhV8YpqX24yWkAAGg9lBUfcPSEU+/mFUpiujIAoO2hrPiA19fvV3WtSz0TbcpMiTE7DgAArcqjZSU1NVUWi6XBberUqQ2O2b9/v2644QZFRETIbrfr3nvvVXV1tSdj+RRnbZ1eXbtPknT34C6yWM4+XTk8JEj5T45U/pMjuQozAMBvePwTbfbs2ZowYYJ7u127du77dXV1GjlypDp27KjVq1fr6NGjGjt2rAzD0NNPP+3paD5h6daDOnLCqXhbqH7WPd7sOAAAtDqPl5XIyEjFxTW90uqKFSu0Y8cOFRQUKCEhQZK0cOFCjRs3TnPmzFFUVJSn43k1wzDcV1e+MztVwYF8awcAaHs8/un31FNPqUOHDurVq5fmzJnT4CuetWvXqlu3bu6iIknDhw+X0+nUpk2bmnw9p9Mph8PR4Oav1u45qq8POhQWHKjb+iWbHQcAAFN4dGTlvvvuU58+fRQTE6MNGzZo2rRp2rt3r1544QVJUnFxsWJjYxs8JyYmRiEhISouLm7yNefNm6dZs2Z5MrbXOD1d+ebMRNnCWQQOANA2NXtkZebMmY1Omv3hbePGjZKkBx54QEOGDFGPHj3061//WosWLdKLL76oo0ePul+vqRNGDcM464mk06ZNU1lZmftWUFDQ3F/BJ+w9UqFPvzkkSbprUKq5YQAAMFGzR1ZycnI0evTocx6Tmpra5P4BAwZIknbt2qUOHTooLi5O69evb3DMsWPHVFNT02jE5TSr1Sqr1drc2D5nyRd7ZRjStV07Ka1ju/M/AQAAP9XssmK322W32y/qh+Xl5UmS4uPrZ7VkZ2drzpw5OnjwoHvfihUrZLValZmZeVE/wx+UVdbo7Y0HJLEIHAAAHjtnZe3atVq3bp2GDh0qm82m3NxcPfDAA7rxxhuVnFx/suiwYcOUkZGhMWPGaP78+SotLdVDDz2kCRMmtOmZQH/L3a+TNXXqGhep7PQOZscBAMBUHisrVqtVb731lmbNmiWn06mUlBRNmDBBDz/8sPuYwMBALV++XPfcc48GDRqksLAw3XbbbVqwYIGnYnm9mjqXXlmTL6l+VOVci8ABANAWeKys9OnTR+vWrTvvccnJyVq2bJmnYvicj74q1sGyKtnbWXVjr4TzPwEAAD/HKmNe5MxF4MYMSJE1KNDkRAAAmI+y4kU27z+mrQXHFRIUoNsHsAgcAAASZcWrnB5V+UWvzrK38//p2QAAXAjKipcoKK3U/35Vv2rvXYNTzQ0DAIAXoax4iVfW5MtlSIMvsatrXNudtg0AwA9RVrzACWet3sqtv2wAi8ABANAQZcUL/E9ugcqdtUrrGKEhl3U0Ow4AAF6FsmKyOpehJWvqT6y9e1AXBQSwCBwAAGeirJjs4x0lKig9qejwYP2yT6LZcQAA8DqUFZO9dGq68m39khUWwiJwAAD8EGXFRNsOlGlDfqmCAiy6MzvV7DgAAHglyoqJXly9R5L08x7xirOFmpwGAADvRFkxSXFZlZZ9eVCSNH5wmslpAADwXpQVk7y6Nl+1LkP9Uture6LN7DgAAHgtyooJTlbX6Y0N+yVJd7MIHAAA50RZMcE7mw/oeGWNktuH66cZsWbHAQDAq1FWWpnLZeilL+qnK48bmKpAFoEDAOCcKCut7LNvD2vP4QpFWoN0a98ks+MAAOD1KCut7MVTi8D9e98ktbMGmZwGAADvR1lpRd8UO7R61xEFWKSxA1PNjgMAgE+grLSi00vrj+gWp6T24SanAQDAN1BWWsmRE069v6VIkjSe6coAAFwwykoreW3dPlXXutQzKVp9kmPMjgMAgM+grLSCqpo6vbZun6T6URWLhenKAABcKMpKK/hwa5GOnKhWvC1U13eLMzsOAAA+hbLiYYZhuE+sHTswVcGBvOUAADQHn5wetnb3UX1TXK6w4ED9qm+y2XEAAPA5lBUPO70I3C1ZibKFB5ucBgAA30NZ8aA9h0/o028OyWKR7hrEdGUAAC4GZcWDlnyRL0m6tmsndbFHmBsGAAAfRVnxkOOV1fr7pgOSpLtZBA4AgIvm0bKSmpoqi8XS4DZ16tQGx/zwcYvFokWLFnkyVqv424YCnayp0xXxUcpO62B2HAAAfJbHL/s7e/ZsTZgwwb3drl27RscsWbJEI0aMcG/bbDZPxzqvyupaZTz2T0nSjtnDFR5y4W9VTZ1Lr6zJl8QicAAA/FgeLyuRkZGKizv3QmjR0dHnPcaX/GPbQRU7qmRvZ9UNPePNjgMAgE/z+DkrTz31lDp06KBevXppzpw5qq6ubnRMTk6O7Ha7+vbtq0WLFsnlcp319ZxOpxwOR4ObNzlzEbg7s1NkDQo0OREAAL7NoyMr9913n/r06aOYmBht2LBB06ZN0969e/XCCy+4j3n88cd17bXXKiwsTJ9++qkefPBBHTlyRI8++miTrzlv3jzNmjXLk7F/lE37jmnrgTKFBAXo9v4sAgcAwI9lMQzDaM4TZs6ced6ykJubq6ysrEb733nnHd188806cuSIOnRo+qTThQsXavbs2SorK2vycafTKafT6d52OBxKSkpSWVmZoqKimvGbnNvFnrPyH69t0kdfFWt03yQ9+cseLZYHAAB/4nA4ZLPZLujzu9kjKzk5ORo9evQ5j0lNTW1y/4ABAyRJu3btOmtZGTBggBwOh0pKShQbG9vocavVKqvV2rzQraSgtFL/3F4siUXgAABoKc0uK3a7XXa7/aJ+WF5eniQpPv7sJ53m5eUpNDRU0dHRF/UzzPTymny5DOknl9p1eVyk2XEAAPALHjtnZe3atVq3bp2GDh0qm82m3NxcPfDAA7rxxhuVnFx/LsfSpUtVXFys7OxshYWFadWqVZo+fbomTpzotaMnZ1NeVaO3cgsksQgcAAAtyWNlxWq16q233tKsWbPkdDqVkpKiCRMm6OGHH3YfExwcrGeffVZTpkyRy+VSWlqaZs+ercmTJ3sqlsf8z8YDOuGsVXrHCA25tKPZcQAA8BseKyt9+vTRunXrznnMiBEjGiwG56vqXIZeXlM/XfnuwV0UEMAicAAAtBSuDdQCPt5RrILSk4oOD9ao3olmxwEAwK9QVlrAi6cWgbu9f7LCQlgEDgCAlkRZ+ZG+PHBcufnHFBxo0Z3ZqWbHAQDA71BWfqTToyo/75Gg2KhQk9MAAOB/KCs/QnFZlZZ/eVBS/dWVAQBAy6Os/AivrM1XrctQvy7t1a2zzew4AAD4JcrKRaqsrtUb6/dLYlQFAABPoqxcpHc2F6rsZI2S24fruisaX8MIAAC0DMrKRXC5DC05dWLtXYNSFcgicAAAeAxl5SL837eHtOdIhSKtQbolK8nsOAAA+DXKykU4PV15dL8ktbN67IoFAABAlJVm+/qgQ1/sOqoAizR2YKrZcQAA8HuUlWZ66dSoyvXd4pUYE25yGgAA/B9lpRkOlzv1wZYiSfVXVwYAAJ5HWWmG19btU3WdS72SopWZEmN2HAAA2gTKygWqqqnTa+v2SWIROAAAWhNl5QJ9uKVIRyuqlWAL1fXd4syOAwBAm0FZuQCGYeilL+pPrB07MFVBgbxtAAC0Fj51L8DaPaX6prhc4SGBGt0v2ew4AAC0KZSVC/DqmnxJ0i2ZibKFBZsbBgCANoaycgE+/+6ILBbprkGcWAsAQGujrFyga7vGKtUeYXYMAADaHMrKBWK6MgAA5qCsXICucZEakNbe7BgAALRJlJWzqKlzue+PHZgii8ViYhoAANouyspZ/HN7sfv+9d3iTUwCAEDbRlk5C8P4/n5IEG8TAABm4VP4LG7omWB2BAAAIMoKAADwcpQVAADg1TxeVpYvX67+/fsrLCxMdrtdo0aNavD4/v37dcMNNygiIkJ2u1333nuvqqurPR0LAAD4iCBPvvg777yjCRMmaO7cubrmmmtkGIa2bdvmfryurk4jR45Ux44dtXr1ah09elRjx46VYRh6+umnPRkNAAD4CI+VldraWt13332aP3++xo8f795/+eWXu++vWLFCO3bsUEFBgRIS6k9oXbhwocaNG6c5c+YoKirKU/EAAICP8NjXQJs3b1ZhYaECAgLUu3dvxcfH6/rrr9f27dvdx6xdu1bdunVzFxVJGj58uJxOpzZt2tTk6zqdTjkcjgY3AADgvzxWVvbs2SNJmjlzph599FEtW7ZMMTExGjJkiEpLSyVJxcXFio2NbfC8mJgYhYSEqLi4uNFrStK8efNks9nct6SkJE/9CgAAwAs0u6zMnDlTFovlnLeNGzfK5apfrn769On65S9/qczMTC1ZskQWi0Vvv/22+/WaWsbeMIyzLm8/bdo0lZWVuW8FBQXN/RUAAIAPafY5Kzk5ORo9evQ5j0lNTVV5ebkkKSMjw73farUqLS1N+/fvlyTFxcVp/fr1DZ577Ngx1dTUNBpxOfM1rFZrc2MDAAAf1eyyYrfbZbfbz3tcZmamrFardu7cqcGDB0uSampqlJ+fr5SUFElSdna25syZo4MHDyo+vv76OytWrJDValVmZmZzowEAAD/ksdlAUVFRmjRpkmbMmKGkpCSlpKRo/vz5kqRbbrlFkjRs2DBlZGRozJgxmj9/vkpLS/XQQw9pwoQJzAQCAACSPLzOyvz58xUUFKQxY8bo5MmT6t+/v1auXKmYmBhJUmBgoJYvX6577rlHgwYNUlhYmG677TYtWLDAk7EAAIAPsRjGmdcX9j0Oh0M2m01lZWUtOhpTWV2rjMf+KUnaMXu4wkM82usAAGhTmvP5zbWBAACAV6OsAAAAr0ZZAQAAXo2yAgAAvBplBQAAeDXKCgAA8GqUFQAA4NUoKwAAwKtRVgAAgFejrAAAAK9GWQEAAF6NsgIAALwaZQUAAHg1ygoAAPBqlBUAAODVKCsAAMCrUVYAAIBXo6wAAACvRlkBAABejbICAAC8WpDZAbxVeEiQ8p8caXYMAADaPEZWAACAV6OsAAAAr0ZZAQAAXo2yAgAAvBplBQAAeDXKCgAA8GqUFQAA4NUoKwAAwKtRVgAAgFejrAAAAK/m8bKyfPly9e/fX2FhYbLb7Ro1alSDxy0WS6PbokWLPB0LAAD4CI9eG+idd97RhAkTNHfuXF1zzTUyDEPbtm1rdNySJUs0YsQI97bNZvNkLAAA4EM8VlZqa2t13333af78+Ro/frx7/+WXX97o2OjoaMXFxXkqCgAA8GEe+xpo8+bNKiwsVEBAgHr37q34+Hhdf/312r59e6Njc3JyZLfb1bdvXy1atEgul+usr+t0OuVwOBrcAACA//JYWdmzZ48kaebMmXr00Ue1bNkyxcTEaMiQISotLXUf9/jjj+vtt9/WJ598otGjR+vBBx/U3Llzz/q68+bNk81mc9+SkpI89SsAAAAvYDEMw2jOE2bOnKlZs2ad85jc3Fx9++23uv322/Xcc89p4sSJkupHRRITE/XEE0/oN7/5TZPPXbhwoWbPnq2ysrImH3c6nXI6ne7tsrIyJScnq6CgQFFRUc35VQAAgEkcDoeSkpJ0/Pjx856r2uxzVnJycjR69OhzHpOamqry8nJJUkZGhnu/1WpVWlqa9u/ff9bnDhgwQA6HQyUlJYqNjW30uNVqldVqdW+f/hqIERYAAHxPeXl5y5cVu90uu91+3uMyMzNltVq1c+dODR48WJJUU1Oj/Px8paSknPV5eXl5Cg0NVXR09AXlSUhIUEFBgSIjI2WxWC7oORfqdOtj1MazeJ9bB+9z6+B9bh28z63HU++1YRgqLy9XQkLCeY/12GygqKgoTZo0STNmzFBSUpJSUlI0f/58SdItt9wiSVq6dKmKi4uVnZ2tsLAwrVq1StOnT9fEiRMbjJ6cS0BAgBITEz31a0iq/134Y/A83ufWwfvcOnifWwfvc+vxxHt9oUuVeHSdlfnz5ysoKEhjxozRyZMn1b9/f61cuVIxMTGSpODgYD377LOaMmWKXC6X0tLSNHv2bE2ePNmTsQAAgA9p9gm2bYnD4ZDNZlNZWRnN3YN4n1sH73Pr4H1uHbzPrccb3muuDXQOVqtVM2bMuOCvpHBxeJ9bB+9z6+B9bh28z63HG95rRlYAAIBXY2QFAAB4NcoKAADwapQVAADg1SgrAADAq1FWzuLZZ59Vly5dFBoaqszMTP3rX/8yO5JfmTdvnvr27avIyEh16tRJN910k3bu3Gl2LL83b948WSwW3X///WZH8UuFhYW644471KFDB4WHh6tXr17atGmT2bH8Sm1trR599FF16dJFYWFh7vW5XC6X2dF82ueff64bbrhBCQkJslgsev/99xs8bhiGZs6cqYSEBIWFhenqq6/W9u3bWy0fZaUJb731lu6//35Nnz5deXl5+slPfqLrr7/+nNc0QvN89tlnmjx5statW6ePP/5YtbW1GjZsmCoqKsyO5rdyc3O1ePFi9ejRw+wofunYsWMaNGiQgoOD9dFHH2nHjh1auHDhBV86BBfmqaee0qJFi/TMM8/o66+/1h/+8AfNnz9fTz/9tNnRfFpFRYV69uypZ555psnH//CHP+iPf/yjnnnmGeXm5iouLk4//elP3dcB9DgDjfTr18+YNGlSg31du3Y1pk6dalIi/3fo0CFDkvHZZ5+ZHcUvlZeXG5deeqnx8ccfG0OGDDHuu+8+syP5nUceecQYPHiw2TH83siRI4277767wb5Ro0YZd9xxh0mJ/I8k47333nNvu1wuIy4uznjyySfd+6qqqgybzWYsWrSoVTIxsvID1dXV2rRpk4YNG9Zg/7Bhw7RmzRqTUvm/srIySVL79u1NTuKfJk+erJEjR+q6664zO4rf+vDDD5WVlaVbbrlFnTp1Uu/evfX888+bHcvvDB48WJ9++qm+/fZbSdLWrVu1evVq/exnPzM5mf/au3eviouLG3wuWq1WDRkypNU+Fz16bSBfdOTIEdXV1Sk2NrbB/tjYWBUXF5uUyr8ZhqEpU6Zo8ODB6tatm9lx/M6bb76pzZs3Kzc31+wofm3Pnj36y1/+oilTpuj3v/+9NmzYoHvvvVdWq1V33nmn2fH8xiOPPKKysjJ17dpVgYGBqqur05w5c/SrX/3K7Gh+6/RnX1Ofi/v27WuVDJSVs7BYLA22DcNotA8tIycnR19++aVWr15tdhS/U1BQoPvuu08rVqxQaGio2XH8msvlUlZWlubOnStJ6t27t7Zv366//OUvlJUW9NZbb+m1117TG2+8oSuvvFJbtmzR/fffr4SEBI0dO9bseH7NzM9FysoP2O12BQYGNhpFOXToUKNWiR/vt7/9rT788EN9/vnnSkxMNDuO39m0aZMOHTqkzMxM9766ujp9/vnneuaZZ+R0OhUYGGhiQv8RHx+vjIyMBvuuuOIKvfPOOyYl8k+/+93vNHXqVI0ePVqS1L17d+3bt0/z5s2jrHhIXFycpPoRlvj4ePf+1vxc5JyVHwgJCVFmZqY+/vjjBvs//vhjDRw40KRU/scwDOXk5Ojdd9/VypUr1aVLF7Mj+aVrr71W27Zt05YtW9y3rKws3X777dqyZQtFpQUNGjSo0fT7b7/9VikpKSYl8k+VlZUKCGj40RUYGMjUZQ/q0qWL4uLiGnwuVldX67PPPmu1z0VGVpowZcoUjRkzRllZWcrOztbixYu1f/9+TZo0yexofmPy5Ml644039MEHHygyMtI9kmWz2RQWFmZyOv8RGRnZ6DygiIgIdejQgfODWtgDDzyggQMHau7cubr11lu1YcMGLV68WIsXLzY7ml+54YYbNGfOHCUnJ+vKK69UXl6e/vjHP+ruu+82O5pPO3HihHbt2uXe3rt3r7Zs2aL27dsrOTlZ999/v+bOnatLL71Ul156qebOnavw8HDddtttrROwVeYc+aA///nPRkpKihESEmL06dOHKbUtTFKTtyVLlpgdze8xddlzli5danTr1s2wWq1G165djcWLF5sdye84HA7jvvvuM5KTk43Q0FAjLS3NmD59uuF0Os2O5tNWrVrV5L/JY8eONQyjfvryjBkzjLi4OMNqtRpXXXWVsW3btlbLZzEMw2idWgQAANB8nLMCAAC8GmUFAAB4NcoKAADwapQVAADg1SgrAADAq1FWAACAV6OsAAAAr0ZZAQAAXo2yAgAAvBplBQAAeDXKCgAA8GqUFQAA4NX+P6OlYxaYLKfVAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.errorbar(x_data, y_data, yerr=ydataerr)\n", + "\n", + "demo_env = envs.training.EpisodeWrapper(env, \n", + " episode_length=1000, \n", + " action_repeat=1)\n", + "\n", + "render_rollout(\n", + " jax.jit(demo_env.reset),\n", + " jax.jit(demo_env.step),\n", + " jax.jit(make_inference_fn(params)),\n", + " demo_env\n", + ")\n", + "\n", + "model_path = '/tmp/trotting_2hz_policy'\n", + "model.save_params(model_path, params)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Study 2: Quadruped Locomotion\n", + "\n", + "As we saw in the imitation learning example, FoPG methods benefit from detailed reward signals. To teach locomotion, we reward the feet based on the Raibert Heuristic. We implement a gait schedule, that incentivizes opposite pairs of legs to move in sync at a fixed frequency. At the beginning of a new scheduled step, we calculate the target position for the feet at the end of the step. \n", + "\n", + "For each foot, we calculate:\n", + "\n", + "$$\n", + "p^* = h_0 + \\frac{\\Delta T}{2} v_0\n", + "$$\n", + "\n", + "Where $p^*$ is x, y component of the foot position target, $h_0$ is the x, y component of the corresponding hip at lift-off, $\\Delta T$ is the scheduled step duration, and $v_0$ is the base velocity at lift-off. \n", + "\n", + "Due to their limited exploration power, FoPG methods benefit greatly from having an good \"initial guess\" of the policy - familiar terminology in Model Predictive Control and Trajectory Optimization. We formulate the problem as [residual learning](https://arxiv.org/abs/1512.03385). Let $\\phi$ be the parameters of a baseline policy that we assume we already have, and let $f$ and $g$ be the neural networks for the learned and baseline policy. We freeze $\\phi$ and learn parameters $\\theta$, for the policy:\n", + "\n", + "$$\n", + "a_t = f(g(x_t; \\phi), x_t; \\theta) + g(x_t; \\phi)\n", + "$$\n", + "\n", + "We use the in-place trotting policy from last section as $\\phi$ and track a 1 m/s velocity target. Since locomotion is stabler at faster trots, you can experiment with $\\phi$ for faster velocity targets!\n", + "\n", + "While a more natural way to \"hotstart\" the learning would be to simply initialize the parameters $\\theta$ as $\\phi$ and use the policy $a_t = f(x_t; \\theta)$, the residual method has been found train more stably in practice." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### The RL Environment" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [], + "source": [ + "def axis_angle_to_quaternion(v: jp.ndarray, theta:jp.float32):\n", + " \"\"\" \n", + " axis angle representation: rotation of theta around v. \n", + " \"\"\" \n", + " return jp.concatenate([jp.cos(0.5*theta).reshape(1), jp.sin(0.5*theta)*v.reshape(3)])\n", + "\n", + "def get_config():\n", + " \"\"\"Returns reward config for anymal quadruped environment.\"\"\"\n", + "\n", + " def get_default_rewards_config():\n", + " default_config = config_dict.ConfigDict(\n", + " dict(\n", + " scales=config_dict.ConfigDict(\n", + " dict(\n", + " tracking_lin_vel = 1.0,\n", + " orientation = -1.0, # non-flat base\n", + " height = 0.5,\n", + " lin_vel_z=-1.0, # prevents the suicide policy\n", + " torque = -0.01,\n", + " feet_pos = -1, # Bad action hard-coding. \n", + " feet_height = -1, # prevents it from just standing still\n", + " joint_velocity = -0.001\n", + " )\n", + " ),\n", + " )\n", + " )\n", + " return default_config\n", + "\n", + " default_config = config_dict.ConfigDict(\n", + " dict(rewards=get_default_rewards_config(),))\n", + "\n", + " return default_config\n", + "\n", + "class DiffAnymal(PipelineEnv):\n", + "\n", + " def __init__(\n", + " self,\n", + " termination_height: float=0.25,\n", + " **kwargs,\n", + " ):\n", + " \n", + " self.target_vel = kwargs.pop('target_vel', 0.75)\n", + " step_k = kwargs.pop('step_k', 25)\n", + " self.baseline_inference_fn = kwargs.pop(\"baseline_inference_fn\")\n", + " physics_steps_per_control_step = 10\n", + " kwargs['n_frames'] = kwargs.get(\n", + " 'n_frames', physics_steps_per_control_step)\n", + " self.termination_height = termination_height\n", + "\n", + " mj_model = mujoco.MjModel.from_xml_path(xml_path)\n", + " self._init_q = mj_model.keyframe('standing').qpos\n", + " self._default_ap_pose = mj_model.keyframe('standing').qpos[7:]\n", + " self.reward_config = get_config()\n", + "\n", + " self.action_loc = self._default_ap_pose\n", + " self.action_scale = jp.array([0.2, 0.8, 0.8] * 4)\n", + " \n", + " self.target_h = self._init_q[2]\n", + "\n", + " sys = mjcf.load_model(mj_model)\n", + " super().__init__(sys=sys, **kwargs)\n", + " \n", + " \"\"\"\n", + " Kinematic references are used for gait scheduling.\n", + " \"\"\"\n", + "\n", + " kinematic_ref_qpos = make_kinematic_ref(\n", + " cos_wave, step_k, scale=0.3, dt=self.dt)\n", + " self.l_cycle = jp.array(kinematic_ref_qpos.shape[0])\n", + "\n", + " # Expand to base state\n", + " kinematic_ref_qpos += self._default_ap_pose\n", + " ref_qs = np.tile(self._init_q.reshape(1, 19), (self.l_cycle, 1))\n", + " ref_qs[:, 7:] = kinematic_ref_qpos\n", + " self.kinematic_ref_qpos = jp.array(ref_qs)\n", + "\n", + " \"\"\"\n", + " Foot tracking\n", + " \"\"\"\n", + " gait_k = step_k * 2\n", + " self.gait_period = gait_k * self.dt\n", + "\n", + " self.step_k = step_k\n", + " self.feet_inds = jp.array([21,28,35,42]) # LF, RF, LH, RH\n", + " self.hip_inds = self.feet_inds - 6\n", + "\n", + " self.pipeline_step = jax.checkpoint(self.pipeline_step,\n", + " policy=jax.checkpoint_policies.dots_with_no_batch_dims_saveable)\n", + " \n", + " def reset(self, rng: jax.Array) -> State:\n", + " rng, key_xyz, key_ang, key_ax, key_q, key_qd = jax.random.split(rng, 6)\n", + "\n", + " qpos = jp.array(self._init_q)\n", + " qvel = jp.zeros(18)\n", + " \n", + " #### Add Randomness ####\n", + " \n", + " r_xyz = 0.2 * (jax.random.uniform(key_xyz, (3,))-0.5)\n", + " r_angle = (jp.pi/12) * (jax.random.uniform(key_ang, (1,)) - 0.5) # 15 deg range\n", + " r_axis = (jax.random.uniform(key_ax, (3,)) - 0.5)\n", + " r_axis = r_axis / jp.linalg.norm(r_axis)\n", + " r_quat = axis_angle_to_quaternion(r_axis, r_angle)\n", + "\n", + " r_joint_q = 0.2 * (jax.random.uniform(key_q, (12,)) - 0.5)\n", + " r_joint_qd = 0.1 * (jax.random.uniform(key_qd, (12,)) - 0.5)\n", + " \n", + " qpos = qpos.at[0:3].set(qpos[0:3] + r_xyz)\n", + " qpos = qpos.at[3:7].set(r_quat)\n", + " qpos = qpos.at[7:19].set(qpos[7:19] + r_joint_q)\n", + " qvel = qvel.at[6:18].set(qvel[6:18] + r_joint_qd)\n", + " \n", + " data = self.pipeline_init(qpos, qvel)\n", + "\n", + " # Ensure you're not sunken into the ground nor above it.\n", + " pen = jp.min(data.contact.dist)\n", + " qpos = qpos.at[2].set(qpos[2] - pen)\n", + " data = self.pipeline_init(qpos, qvel)\n", + "\n", + " state_info = {\n", + " 'rng': rng,\n", + " 'steps': 0.0,\n", + " 'reward_tuple': {\n", + " 'tracking_lin_vel': 0.0,\n", + " 'orientation': 0.0,\n", + " 'height': 0.0,\n", + " 'lin_vel_z': 0.0,\n", + " 'torque': 0.0,\n", + " 'joint_velocity': 0.0,\n", + " 'feet_pos': 0.0,\n", + " 'feet_height': 0.0\n", + " },\n", + " 'last_action': jp.zeros(12), # from MJX tutorial.\n", + " 'baseline_action': jp.zeros(12),\n", + " 'xy0': jp.zeros((4, 2)),\n", + " 'k0': 0.0,\n", + " 'xy*': jp.zeros((4, 2))\n", + " }\n", + "\n", + " x, xd = data.x, data.xd\n", + " _obs = self._get_obs(data.qpos, x, xd, state_info) # inner obs; to trotter\n", + " \n", + " action_key, key = jax.random.split(state_info['rng'])\n", + " state_info['rng'] = key\n", + " next_action, _ = self.baseline_inference_fn(_obs, action_key)\n", + "\n", + " obs = jp.concatenate([_obs, next_action])\n", + "\n", + " reward, done = jp.zeros(2)\n", + " metrics = {}\n", + " for k in state_info['reward_tuple']:\n", + " metrics[k] = state_info['reward_tuple'][k]\n", + " state = State(data, obs, reward, done, metrics, state_info)\n", + " return jax.lax.stop_gradient(state)\n", + "\n", + " def step(self, state: State, action: jax.Array) -> State:\n", + "\n", + " action = jp.clip(action, -1, 1)\n", + "\n", + " cur_base = state.obs[-12:]\n", + " action += cur_base\n", + " state.info['baseline_action'] = cur_base\n", + "\n", + " action = self.action_loc + (action * self.action_scale)\n", + "\n", + " data = self.pipeline_step(state.pipeline_state, action)\n", + " \n", + " # observation data\n", + " x, xd = data.x, data.xd\n", + " obs = self._get_obs(data.qpos, x, xd, state.info)\n", + "\n", + " # Terminate if flipped over or fallen down.\n", + " done = 0.0\n", + " done = jp.where(x.pos[0, 2] < self.termination_height, 1.0, done)\n", + " up = jp.array([0.0, 0.0, 1.0])\n", + " done = jp.where(jp.dot(math.rotate(up, x.rot[0]), up) < 0, 1.0, done)\n", + "\n", + " #### Foot Position Reference Updating ####\n", + "\n", + " # Detect the start of a new step\n", + " s = state.info['steps']\n", + " step_num = s // (self.step_k)\n", + " even_step = step_num % 2 == 0\n", + " new_step = (s % self.step_k) == 0\n", + " new_even_step = jp.logical_and(new_step, even_step)\n", + " new_odd_step = jp.logical_and(new_step, jp.logical_not(even_step))\n", + "\n", + " # Apply railbert heuristic to calculate target foot position, after step\n", + " hip_xy = data.geom_xpos[self.hip_inds][:,:2] # 4 x 2\n", + " v_body = data.qvel[0:2]\n", + " step_period = self.gait_period/2\n", + " raibert_xy = hip_xy + (step_period/2) * v_body\n", + "\n", + " # Update. \n", + " cur_tars = state.info['xy*']\n", + " i_RFLH = jp.array([1, 2])\n", + " i_LFRH = jp.array([0, 3])\n", + " feet_xy = data.geom_xpos[self.feet_inds][:,:2]\n", + " \n", + " # With the trotting gait, we will move one pair of opposite legs, \n", + " # and keep the other pair fixed in place.\n", + " case_c1 = raibert_xy.at[i_LFRH].set(feet_xy[i_LFRH]) \n", + " case_c2 = raibert_xy.at[i_RFLH].set(feet_xy[i_RFLH])\n", + " xy_tars = jp.where(new_even_step, case_c1, cur_tars)\n", + " xy_tars = jp.where(new_odd_step, case_c2, xy_tars)\n", + " state.info['xy*'] = xy_tars\n", + "\n", + " # Save timestep and location at start of step.\n", + " state.info['k0'] = jp.where(new_step,\n", + " state.info['steps'],\n", + " state.info['k0'])\n", + " state.info['xy0'] = jp.where(new_step, \n", + " feet_xy,\n", + " state.info['xy0'])\n", + "\n", + " # reward\n", + " reward_tuple = {\n", + " 'tracking_lin_vel': (\n", + " self._reward_tracking_lin_vel(jp.array([self.target_vel, 0, 0]), x, xd)\n", + " * self.reward_config.rewards.scales.tracking_lin_vel\n", + " ),\n", + " 'orientation': (\n", + " self._reward_orientation(x)\n", + " * self.reward_config.rewards.scales.orientation\n", + " ),\n", + " 'lin_vel_z': (\n", + " self._reward_lin_vel_z(xd)\n", + " * self.reward_config.rewards.scales.lin_vel_z\n", + " ),\n", + " 'height': (\n", + " self._reward_height(data.qpos) \n", + " * self.reward_config.rewards.scales.height\n", + " ),\n", + " 'torque': (\n", + " self._reward_action(data.qfrc_actuator)\n", + " * self.reward_config.rewards.scales.torque\n", + " ),\n", + " 'joint_velocity': (\n", + " self._reward_joint_velocity(data.qvel)\n", + " * self.reward_config.rewards.scales.joint_velocity\n", + " ),\n", + " 'feet_pos': (\n", + " self._reward_feet_pos(data, state)\n", + " * self.reward_config.rewards.scales.feet_pos\n", + " ),\n", + " 'feet_height': (\n", + " self._reward_feet_height(data, state.info)\n", + " * self.reward_config.rewards.scales.feet_height\n", + " )\n", + " }\n", + " \n", + " reward = sum(reward_tuple.values())\n", + "\n", + " # state management\n", + " state.info['reward_tuple'] = reward_tuple\n", + " state.info['last_action'] = action\n", + "\n", + " for k in state.info['reward_tuple'].keys():\n", + " state.metrics[k] = state.info['reward_tuple'][k]\n", + "\n", + " # next action\n", + " action_key, key = jax.random.split(state.info['rng'])\n", + " state.info['rng'] = key\n", + " next_action, _ = self.baseline_inference_fn(obs, action_key)\n", + " obs = jp.concatenate([obs, next_action])\n", + "\n", + " state = state.replace(\n", + " pipeline_state=data, obs=obs, reward=reward,\n", + " done=done)\n", + " return state\n", + "\n", + " def _get_obs(self, qpos: jax.Array, x: Transform, xd: Motion,\n", + " state_info: Dict[str, Any]) -> jax.Array:\n", + "\n", + " inv_base_orientation = math.quat_inv(x.rot[0])\n", + " local_rpyrate = math.rotate(xd.ang[0], inv_base_orientation)\n", + "\n", + " obs_list = []\n", + " # yaw rate\n", + " obs_list.append(jp.array([local_rpyrate[2]]) * 0.25)\n", + " # projected gravity\n", + " obs_list.append(\n", + " math.rotate(jp.array([0.0, 0.0, -1.0]), inv_base_orientation))\n", + " # motor angles\n", + " angles = qpos[7:19]\n", + " obs_list.append(angles - self._default_ap_pose)\n", + " # last action\n", + " obs_list.append(state_info['last_action'])\n", + " # base action\n", + " cur_base = self.kinematic_ref_qpos[jp.array(state_info['steps']%self.l_cycle, int)]\n", + " obs_list.append(cur_base)\n", + "\n", + " obs = jp.clip(jp.concatenate(obs_list), -100.0, 100.0)\n", + "\n", + " return obs\n", + "\n", + " # ------------ reward functions----------------\n", + " def _reward_tracking_lin_vel(\n", + " self, commands: jax.Array, x: Transform, xd: Motion) -> jax.Array:\n", + " # Tracking of linear velocity commands (xy axes)\n", + " local_vel = math.rotate(xd.vel[0], math.quat_inv(x.rot[0]))\n", + " lin_vel_error = jp.sum(jp.square(commands[:2] - local_vel[:2]))\n", + " lin_vel_reward = jp.exp(\n", + " -lin_vel_error\n", + " )\n", + " return lin_vel_reward\n", + " def _reward_orientation(self, x: Transform) -> jax.Array:\n", + " # Penalize non flat base orientation\n", + " up = jp.array([0.0, 0.0, 1.0])\n", + " rot_up = math.rotate(up, x.rot[0])\n", + " return jp.sum(jp.square(rot_up[:2]))\n", + " def _reward_lin_vel_z(self, xd: Motion) -> jax.Array:\n", + " # Penalize z axis base linear velocity\n", + " return jp.clip(jp.square(xd.vel[0, 2]), 0, 10)\n", + " def _reward_joint_velocity(self, qvel):\n", + " return jp.clip(jp.sqrt(jp.sum(jp.square(qvel[6:]))), 0, 100)\n", + " def _reward_height(self, qpos) -> jax.Array:\n", + " return jp.exp(-jp.abs(qpos[2] - self.target_h)) # Not going to be > 1 meter tall.\n", + " def _reward_action(self, action) -> jax.Array:\n", + " return jp.sqrt(jp.sum(jp.square(action)))\n", + " def _reward_feet_pos(self, data, state): \n", + " dt = (state.info['steps'] - state.info['k0']) * self.dt # scalar\n", + " step_period = self.gait_period / 2\n", + " xyt = state.info['xy0'] + (state.info['xy*'] - state.info['xy0']) * (dt/step_period)\n", + "\n", + " feet_pos = data.geom_xpos[self.feet_inds][:, :2]\n", + "\n", + " rews = jp.sum(jp.square(feet_pos - xyt), axis=1) \n", + " rews = jp.clip(rews, 0, 10)\n", + " return jp.sum(rews)\n", + " \n", + " def _reward_feet_height(self, data, state_info):\n", + " \"\"\" \n", + " Feet height tracks rectified sine waves \n", + " \"\"\"\n", + " h_tar = 0.1\n", + " t = state_info['steps'] * self.dt\n", + " offset = self.gait_period/2\n", + " ref1 = jp.sin((2*jp.pi/self.gait_period)*t) # RF and LH feet\n", + " ref2 = jp.sin((2*jp.pi/self.gait_period)*(t - offset)) # LF and RH\n", + " \n", + " ref1, ref2 = ref1 * h_tar, ref2 * h_tar\n", + " h_tars = jp.array([ref2, ref1, ref1, ref2])\n", + " h_tars = h_tars.clip(min=0, max=None) + 0.02 # offset height of feet.\n", + " \n", + " feet_height = data.geom_xpos[self.feet_inds][:,2]\n", + " errs = jp.clip(jp.square(feet_height - h_tars), 0, 10)\n", + " return jp.sum(errs)\n", + " \n", + "envs.register_environment('anymal', DiffAnymal)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Residual Learning via FoPG's\n", + "Takes 15 minutes on a NVIDIA 3060 TI GPU" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [], + "source": [ + "# Reconstruct the trotting inference function\n", + "\n", + "make_networks_factory = functools.partial(\n", + " apg_networks.make_apg_networks,\n", + " hidden_layer_sizes=(256, 128)\n", + ")\n", + "\n", + "nets = make_networks_factory(observation_size=47,\n", + " action_size=12,\n", + " preprocess_observations_fn=running_statistics.normalize)\n", + "\n", + "make_inference_fn = apg_networks.make_inference_fn(nets)\n", + "\n", + "# Configure locomotion training\n", + "make_networks_factory = functools.partial(\n", + " apg_networks.make_apg_networks,\n", + " hidden_layer_sizes=(128, 64)\n", + ")\n", + "\n", + "epochs = 499\n", + "\n", + "train_fn = functools.partial(apg.train,\n", + " episode_length=1000,\n", + " policy_updates=epochs,\n", + " horizon_length=32,\n", + " num_envs=64,\n", + " learning_rate=2.5e-4,\n", + " num_eval_envs=64,\n", + " num_evals=10 + 1,\n", + " normalize_observations=True,\n", + " network_factory=make_networks_factory)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [], + "source": [ + "x_data = []\n", + "y_data = []\n", + "ydataerr = []\n", + "times = [datetime.now()]\n", + "\n", + "def progress(it, metrics):\n", + " times.append(datetime.now())\n", + " x_data.append(it)\n", + " y_data.append(metrics['eval/episode_reward'])\n", + " ydataerr.append(metrics['eval/episode_reward_std'])\n", + "\n", + "model_path = '/tmp/trotting_2hz_policy'\n", + "params = model.load_params(model_path)\n", + "baseline_inference_fn = make_inference_fn(params)\n", + "\n", + "env_kwargs = dict(target_vel=1, step_k=13, \n", + " baseline_inference_fn=baseline_inference_fn)\n", + "\n", + "env = envs.get_environment(\"anymal\", **env_kwargs)\n", + "eval_env = envs.get_environment(\"anymal\", **env_kwargs)\n", + "\n", + "make_inference_fn, params, _= train_fn(environment=env,\n", + " progress_fn=progress,\n", + " eval_env=eval_env)" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjMAAAGdCAYAAADnrPLBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABHXUlEQVR4nO3deVzUdeLH8ddwjYCAKMqhKGiYIpaKRx6tmYqV1nZsVpZpmT9djzTsMtty/SWWlblZa2uHZ1bbllu7VorVzzJNEY88UjRF8UBSieEeYL6/P8wpvA+GLzO8n4/HPGK+85nhzaTM28/38/1+LYZhGIiIiIi4KS+zA4iIiIhcDpUZERERcWsqMyIiIuLWVGZERETEranMiIiIiFtTmRERERG3pjIjIiIibk1lRkRERNyaj9kBqoPD4eDQoUMEBQVhsVjMjiMiIiIXwDAM8vPziYqKwsvr7PMvtaLMHDp0iOjoaLNjiIiIyCXIysqiSZMmZ328VpSZoKAg4MSbERwcbHIaERERuRA2m43o6Gjn5/jZ1Ioyc3LXUnBwsMqMiIiImznfEhEtABYRERG3pjIjIiIibk1lRkRERNyayoyIiIi4NZUZERERcWsqMyIiIuLWVGZERETEranMiIiIiFtTmRERERG3pjIjIiIibk1lRkRERNyayoyIiIi4NZUZERERcWsqMyIiInJJiuzlxDy5lJgnl1JkLzcth8qMiIiIuDWVGREREXFrKjMiIiJy2YrtFaZ9b5eWmfLycp5++mliY2Px9/enefPmTJkyBYfD4RxjGAaTJ08mKioKf39/rrvuOrZt21bpdUpLSxk7dixhYWEEBgZyyy23cODAAVdGFxERqVY1Zf3JmZSWV7D3aCHfZPzM4rX7eeGLHYx9byP3zPneOeaTzYdMy+fjyhd/4YUXeOONN5g/fz5t2rRh/fr1PPDAA4SEhDBu3DgApk+fzowZM5g3bx4tW7bkueeeo2/fvuzcuZOgoCAAxo8fz3/+8x/ef/99GjRowIQJExgwYADp6el4e3u78kcQERHxeGUVDg79UsyB3GIO5BaRdfzEfw/kFpOVW8QRW+l5XyM7r6Qakp6ZS8vMmjVr+OMf/0j//v0BiImJ4b333mP9+vXAiVmZmTNnMmnSJG6//XYA5s+fT3h4OIsXL2bEiBHk5eXx9ttvs3DhQvr06QPAokWLiI6OZsWKFfTr18+VP4KIiIjbK69wcDivxFlOTpaWA7+WlmxbCQ7j3K/h7+tNdH1/moQGEB164r8Ng/wY/8FmAMb1vqIafpIzc2mZ6dGjB2+88QYZGRm0bNmSzZs3s2rVKmbOnAnA3r17yc7OJikpyfkcq9VKz549Wb16NSNGjCA9PZ2ysrJKY6KiokhISGD16tVnLDOlpaWUlv7WIm02m+t+SBEREZNVOAyO2H4tK8d/Kysni8vhvBIqztNWrD5eNAn1J7p+AE1CT5aWk1/7Uz/QD4vFUuk5J3aHnSgzpz5WnVxaZp544gny8vJo1aoV3t7eVFRUMHXqVO655x4AsrOzAQgPD6/0vPDwcPbt2+cc4+fnR2ho6GljTj7/VNOmTeOvf/1rVf84IiLiJors5cQ/swyA7VP6EeDn0o87l3M4DH4uKK1cVI4Xc+CXE/cP/VJMWcW5y4qf94my0vhkUfl1lqVJqD/RoQGE1T29rLgLl/7f/eCDD1i0aBGLFy+mTZs2bNq0ifHjxxMVFcWQIUOc40598wzDOO8beq4xEydOJDk52XnfZrMRHR19GT+JiIhI9ZnzzR6O2Eqd61YO5hZjr3Cc8zk+XpZfi4r/72ZUfistDeta8fJyz7JyPi4tM4899hhPPvkkd999NwBt27Zl3759TJs2jSFDhhAREQGcmH2JjIx0Pi8nJ8c5WxMREYHdbic3N7fS7ExOTg7dunU74/e1Wq1YrVZX/VgiIiJVKrfQzuyVPznvz1yx67Qx3l4WIkPqVCoqv98tFB5cB+9qLisBfj5kPt+/Wr/nmbi0zBQVFeHlVfnob29vb+eh2bGxsURERJCamkr79u0BsNvtrFy5khdeeAGAxMREfH19SU1NZeDAgQAcPnyYrVu3Mn36dFfGFxERcalfiuy8+e0e5n2XSeHvztPyx3ZRNGsQ6Fxo2yTUn8iQOvh46/RwZ+LSMnPzzTczdepUmjZtSps2bdi4cSMzZszgwQcfBE7sXho/fjwpKSnExcURFxdHSkoKAQEBDBo0CICQkBCGDRvGhAkTaNCgAfXr1+fRRx+lbdu2zqObRERE3MkvRXbe+nYv81ZnUlB64pwyrSKC2JGdD8C029u6/Tqf6uTSd2rWrFn85S9/YdSoUeTk5BAVFcWIESN45plnnGMef/xxiouLGTVqFLm5uXTp0oXly5c7zzED8Morr+Dj48PAgQMpLi6md+/ezJs3T+eYERERt/JLkZ23V+1l7ne/lZjWkcGM7xNHjysa0ObZ5SYndE8WwzDOc2S5+7PZbISEhJCXl0dwcLDZcURExMVq2tFMeUVlvL1qD3O/yyT/dzMx4/u0JCk+HC8vS43LXBNc6Oe33ikREREXOXuJiSMpPsJjjy6qbiozIiIiVSyvqIy3v9vL3FV7VWKqgcqMiIhIFckrLuOdVXt557u95JecKDFXhp8oMf3anLvE1JTDnN2RyoyIiMhlyisuY+53e3l7VeUSM65PHDecp8TI5VOZERGRc9LC1LM7U4lpGV6Xcb1bcmOCSkx10Z9IERGRi2QrKWPuqkzeXrUHm0qM6VRmRERELpCtpIx532Xy1re/lZi4RnUZ1yeOmxIiVWJMojIjIiJyHvklZcw9pcRc0agu43rHcVPbyGq/JpJUpjIjIiJyFvknZ2JW7SWvuAxQiamJVGZEREROkV9SxvzVmbz57W8lpkXDQMb1aUl/lZgaR2VGRETkV/klZSxYs483v93DL0W/lZiHe8cx4KoolZgaSmVGRERqvYLS8l9nYn4rMc0bBjJOJcYtqMyIiEitpRLjGVRmRESk1ikoLWfBmkze/GYPuSdLTNiJ3Uk3X60S425UZkREpNYoLC1nwZp9zPnmJ2eJiQ0L5OHeV3DzVVH4eHuZnFAuhcqMiIh4vLOVmLHXX8EtV6vEuDuVGRER8Whvf7uXuaszOV5oByCmQQAP945TifEgKjMiIuIxiuzl7MjOZ3PWL85tL6dmACdKzNjr4/hjO5UYT6MyIyIibscwDI7YStl+OI8fD+ez/bCNHw/Z2HusEMOoPDa6vj/jerfkVpUYj6UyIyIiNVpZhYPdOQX8eNjG9kM2fsw+8d+Ta19OFVbXSquIuqzafQyApWN7EOzvV52RpZqpzIiISI3xS5H9xCzL4fwTxeWwjV05+ZRVGKeN9fay0DwskPioYFpHnrwF0SioDkX2cuKfWQag2ZhaQGVGRESqncNhsO94ET8etv0243LYxqG8kjOOD7L60Doy+NfiEkR8ZAhx4XWp4+tdzcmlJlKZERERlyqyl7Mz+9d1Lb8Wl53Z+RTaK844Prq+P60jgp0zLvGRwTQJ9cdi0Yns5MxUZkREpEqcXJT742Eb23+9nW1RLoDVx4srI4IqFZdWkUEE1/Gt/vDi1lRmRESq0e/Xcmyf0o8AP/f8NXyxi3IbBlmda1rif51tiQ0L1HoWqRLu+bdIRERM8dTHW8g4UnDORbktGgY6F+TG//rfhkFWE9JKbaEyIyIi57QzO9/59b83HXJ+HWT1oXVUsHOmpXVksBbliilUZkRE5Kz2Hi3koQXrnfdH92rB1U3q0VqLcqUGUZkREZEzOvhLMfe9tZZjBXbnttG9rnDbdT7iubTySkRETvNzfin3vbWWg78UExsWaHYckXNSmRERkUryisoY/PZa9h4tpHE9f94akmh2JJFz0lyhiIg4FZaWM3TeOnZk59MwyMq7D3WhUbCORJKaTTMzIiICQElZBcMXrGfj/l8I8fdl4bDOxGgXk7gBlRkREaGswsGYxRtY/dMxAv28mf9gZ1pFBJsdS+SCqMyIiNRyFQ6DRz/czIofc7D6ePHWkE60i65ndiyRC+byMnPw4EHuu+8+GjRoQEBAAO3atSM9Pd35uGEYTJ48maioKPz9/bnuuuvYtm1bpdcoLS1l7NixhIWFERgYyC233MKBAwdcHV1ExOMZhsHT/97KJ5sO4eNlYfZ9HejaooHZsUQuikvLTG5uLt27d8fX15fPP/+c7du38/LLL1OvXj3nmOnTpzNjxgxee+010tLSiIiIoG/fvuTn/3bGyfHjx7NkyRLef/99Vq1aRUFBAQMGDKCi4sxXXBURkfMzDINpn+/gvXX78bLAzLvbcX2rcLNjiVw0lx7N9MILLxAdHc3cuXOd22JiYpxfG4bBzJkzmTRpErfffjsA8+fPJzw8nMWLFzNixAjy8vJ4++23WbhwIX369AFg0aJFREdHs2LFCvr16+fKH0FExGO99tVu5nyzB4Bpt7dlwFVRJicSuTQunZn59NNP6dixI3feeSeNGjWiffv2vPnmm87H9+7dS3Z2NklJSc5tVquVnj17snr1agDS09MpKyurNCYqKoqEhATnmFOVlpZis9kq3URE5DfvrNrLy6kZAPxlQDx3dWpqciKRS+fSMrNnzx5mz55NXFwcy5YtY+TIkTz88MMsWLAAgOzsbADCwytPa4aHhzsfy87Oxs/Pj9DQ0LOOOdW0adMICQlx3qKjo6v6RxMRcVv/XJ/FlP9uB2B8nziG9Yg1OZHI5XFpmXE4HHTo0IGUlBTat2/PiBEjGD58OLNnz6407tQLlRmGcd6Ll51rzMSJE8nLy3PesrKyLu8HERHxEEt/OMyTH/0AwEM9YhnXO87kRCKXz6VlJjIykvj4+ErbWrduzf79+wGIiIgAOG2GJScnxzlbExERgd1uJzc396xjTmW1WgkODq50ExGp7b7emcP4DzbiMODuTtFM6t9aV70Wj+DSMtO9e3d27txZaVtGRgbNmjUDIDY2loiICFJTU52P2+12Vq5cSbdu3QBITEzE19e30pjDhw+zdetW5xgRETm3tXuOMXJhOmUVBgOuimTqbW1VZMRjuPRopkceeYRu3bqRkpLCwIEDWbduHXPmzGHOnDnAid1L48ePJyUlhbi4OOLi4khJSSEgIIBBgwYBEBISwrBhw5gwYQINGjSgfv36PProo7Rt29Z5dJOIiJzdDwd+Ydj89ZSWO+jdqhGv3NUOby/PLjIBfj5kPt/f7BhSTVxaZjp16sSSJUuYOHEiU6ZMITY2lpkzZ3Lvvfc6xzz++OMUFxczatQocnNz6dKlC8uXLycoKMg55pVXXsHHx4eBAwdSXFxM7969mTdvHt7e3q6MLyLi9jKO5HP/O+soKC2na/MGvH5vB3y9dfJ38SwWwzAMs0O4ms1mIyQkhLy8PK2fERFTFdnLiX9mGQDbp/QjwM91/6bcd6yQO99YQ05+KVdH1+Pdh7pQ13rx3686M4v83oV+fquei4h4oMN5xdz71lpy8ktpFRHE/Ac6XVKREXEHKjMiIh7mWEEp9721lgO5xcQ0CGDBsM7UC/AzO5aIy6jMiIh4kLziMu5/Zx0//VxIVEgdFj3UhUZBdcyOJeJSKjMiIh6iyF7Og/PS2HbIRlhdPxY91IUmoQFmxxJxOe1AFRHxAKXlFYxYmE76vlyC6/iw4MEuNG9Yt0peW4c5S02nmRkRETdXXuHg4fc28u2uowT4eTPvwc7ER+nITak9VGZEBDixiyLmyaXEPLmUInu52XHkAjkcBo//6weWbTuCn48Xb97fkQ5NQ8//RBEPojIjIuKmDMPg2U+38fHGg3h7WXh9UAe6XxFmdiyRaqcyIyLipl5ctpOF3+/DYoEZA6+mb/yZL74r4ulUZkRE3NDf/283f/+/nwCYemtb/tiuscmJRMyjMiMi4mYWrslk+hc7AXjqplYM6tLU5EQi5lKZERFxIx9vOMBfPtkGwMPXX8H//KGFyYlEzKcyIyLiJr7Yms1j//oBgKHdYnikb0uTE4nUDCozIiJu4NtdP/PwexupcBjcmdiEZwbEY7FYzI4lUiOozIiI1HDrM4/zPwvSsVc4uKltBM/fcRVeXioyIiepzIiI1GBbD+bxwNw0issq6NmyITPvao+3ioxIJSozIiI11O6cfO5/Zx35peV0jqnPG/cl4uejX9sip9LfChFxW558CYas40Xc99Y6jhfauapJCG8P7Yi/n7fZsURqJJUZEZEa5oithHvfWku2rYSW4XWZ/0Bngur4mh1LpMZSmRERqUGOF9q576217D9eRNP6ASwa1oXQQD+zY4nUaCozIiI1RH5JGUPeWceunAIiguvw7kNdaBRcx+xYIjWeyoyISA1QbK9g2Pz1bDmYR/1APxY91IXo+gFmxxJxCyozIiIms5c7GLkonXV7jxNUx4cFD3bmikZ1zY4l4jZUZkRETFRe4WD8BxtZmfEz/r7ezB3aiYTGIWbHEnErPmYHEBGprRwOg4mfbOGzLdn4eXsx5/5EOsbUNzuWiNvRzIyIiEme/2IHH6YfwNvLwqv3tOfauIZmRxJxSyozIiImWfT9fgBe/NNV3JAQYXIaEfelMiMiUo0Mw6h0/3//2IbbOzQxKY2IZ1CZERGpJnlFZTzywWbn/Uf6xDG4a4x5gUQ8hBYAi4hUg/WZxxn3/iYO/lLs3Db8D81NTCTiOVRmRERcqMJh8MbKn5iRmkGFwyC6vj9Zx4vP/0QRuWDazSQi4iJHbCUMfnstLy7bSYXD4I/tovhoZDezY4l4HM3MiIi4wNc7c5jwz80cL7Tj7+vNlD+24U+JTSguqzA7mojHUZkREalC9nIH07/YwVur9gLQOjKYWfe01+UJRFxIZUZEpIpkHi3k4fc38sOBPACGdovhyRtbUcfX2+RkIp6t2tbMTJs2DYvFwvjx453bDMNg8uTJREVF4e/vz3XXXce2bdsqPa+0tJSxY8cSFhZGYGAgt9xyCwcOHKiu2CIiF+STTQfp/+q3/HAgj3oBvswZnMjkW9qoyIhUg2opM2lpacyZM4errrqq0vbp06czY8YMXnvtNdLS0oiIiKBv377k5+c7x4wfP54lS5bw/vvvs2rVKgoKChgwYAAVFdrvLCLmKywt59EPNzPu/U0U2ivoHFOfzx6+lqQ2OqOvSHVxeZkpKCjg3nvv5c033yQ0NNS53TAMZs6cyaRJk7j99ttJSEhg/vz5FBUVsXjxYgDy8vJ4++23efnll+nTpw/t27dn0aJFbNmyhRUrVrg6ukitsvqnY86vZ6RmkFdUZmIa97DtUB43v7aKf6UfwMsC43rHsXh4F6Lq+ZsdTaRWcXmZGT16NP3796dPnz6Vtu/du5fs7GySkpKc26xWKz179mT16tUApKenU1ZWVmlMVFQUCQkJzjFnUlpais1mq3QTkbPbfsjGuPc3Ou+/9e1eekz/ite/3k1habmJyWomwzCYvzqT215fzZ6fC4kIrsPi4dfwSN+W+HjrjBci1c2lC4Dff/99NmzYQFpa2mmPZWdnAxAeHl5pe3h4OPv27XOO8fPzqzSjc3LMyeefybRp0/jrX/96ufFFaoVDvxTzwLx1FJb+tus2rlFdduUU8OKyncz9bi9jel3BPV2aYvXR+o/cQjuPf/QDqduPANCndSOm/+lq6gf6mZxMpPZy2T8hsrKyGDduHIsWLaJOnTpnHWexWCrdNwzjtG2nOt+YiRMnkpeX57xlZWVdXHiRWiKvuIyhc9dxxFZa6dDhj0d1Y+Zd7WhaP4CjBXYm/2c717+0kg/XZ1HhMM7xip5t3d7j3PTqt6RuP4KftxfP3hzPm/d3VJERMZnLykx6ejo5OTkkJibi4+ODj48PK1eu5NVXX8XHx8c5I3PqDEtOTo7zsYiICOx2O7m5uWcdcyZWq5Xg4OBKNxGprLS8ghEL15NxpIDwYCtv3NfB+Zi3l4Vb2zfmywk9ee7WBBoFWTn4SzGP/esH+s38hs+3HD7t6s+erMJhMHNFBnfPWcPhvBKahwXy8ahuPNA99rz/+BIR13NZmenduzdbtmxh06ZNzlvHjh2599572bRpE82bNyciIoLU1FTnc+x2OytXrqRbtxOn+05MTMTX17fSmMOHD7N161bnGBG5eA6HwWMf/sD3e45T1+rD3KGdz7ho1dfbi/uuacbKx3ox8cZW1AvwZXdOAX9+dwO3vPYd32T87PGl5nBeMYPe/J6ZK3bhMOCODk34z9geJDQOMTuaiPzKZWtmgoKCSEhIqLQtMDCQBg0aOLePHz+elJQU4uLiiIuLIyUlhYCAAAYNGgRASEgIw4YNY8KECTRo0ID69evz6KOP0rZt29MWFIvIhZu+bCefbj6Ej5eF2fd1ID4qmCL72Rf6+vt5M6JnC+7p0pS3vtnDW6v2suVgHve/s44usfV5/IZWJDYLPevz3dWK7Ud47F+byS0qI9DPm+duS+C29k3MjiUipzD1DMCPP/44xcXFjBo1itzcXLp06cLy5csJCgpyjnnllVfw8fFh4MCBFBcX07t3b+bNm4e3txYiilyKBWsyeWPlTwC8cMdVXBvX8IKfG1zHl+SkK7m/Wwx///onFn2/j7V7j3PH7NX0ad2ICUlX0jrS/XfrlpZX8PznO5j7XSYACY2DmXVPB2LDAs0NJiJnZDE8fY4YsNlshISEkJeXp/UzUqst35bNyEXpOAyY0LclY3vHOR8rspcT/8wyALZP6UeA3/n/rXPwl2JeXbGLD9OzcBhgscAtV0fxSJ+WxFTDB/+lZD6fPT8XMPa9jWw7dOKUDsN6xPL4DVdW2ZFcrsgs4qku9PNbJ0QQqSU27M/l4fc34jDgns7RjLn+ist+zcb1/HnhT1eRmtyT/ldFYhjwyaZD9JmxkqeWbCE7r6QKklefj9IPMGDWKrYdslE/0I+5QzvxlwHxOiRdpIZTmRGpBTKPFvLQ/PWUlDnodWVD/vePCVV6FE6LhnV5fVAH/ju2B9dd2ZByh8Hitfvp+eLXTF26neOF9ir7Xq5QUFrOIx9sYsKHmymyV9C1eQM+H3ctvVo1MjuaiFwAzW+KeLhjBaUMmbuO44V22jYO4bVBHVx2ltqExiHMe6Az6/Ye58VlO0jLzOXNb/fy3roshl/bnGHXxlLXWrN+7Ww5kMfY9zaQeawIby8Lj/SJ48/XXYG3lw65FnEXmpkR8WDF9goenL+efceKiK7vzztDOxFYDWWic2x9/jmiK3Mf6ER8ZDAFpeW8siKDP0z/mre+3UNJmfkXijUMg7dX7eX22d+ReayIxvX8+eB/rmHM9XEqMiJupmb9E0lEqkyFw2DsexvZnPUL9QJ8mfdAZxoGWavt+1ssFnpd2YiecQ35bOthZizPYM/RQp5b+iNvr9rLuN5x/CmxiSnXMjpWUMpj//qBr3bkANCvTTjT77iakADfas8iIpdPZUbEAxmGwbOfbmXFj0fw8/Hirfs70qJh3fM/0QW8vCwMuCqKG9pE8NGGA8xcsYvDeSU8+fEW/vHNHpL7tqR/20i8qmk2ZPVPRxn//iZy8kvx8/HiLwPiua9LU53JV8SNaTeTiAd6Y+UeFn2/H4sF/nZXOzrG1Dc7Ej7eXtzVqSlfP3odfxkQT/1AP/YeLWTsexsZMGsVX+/IcenZhMsrHMxYvpN731pLTv6Ja1F9Mro7g69ppiIj4uY0MyPiYT7ZdJAXvtgBwF/6x3Nj20iTE1VWx9ebYT1iuatTNO+s2sub3+xh+2EbD8xLo1NMKI/1a0Xn2KotXwd/KWb8+xtJyzxxnbe7O0XzzM3xOseLiIfQzIyIB1n901Ee/XAzAA/1iOXBHrEmJzq7ulYfHu4dxzeP92LEH5pj9fEiLTOXgf9Yw5B31rH1YF6VfJ8vtmZz09++JS0zlyCrD7Puac/zd1ylIiPiQfS3WcRD7Mi2MWJBOmUVBv2viuSpm1qbHemChAb6MfGm1jzQPZZZX+3ig7QsVmb8zMqMn+nfNpLkpJaXtN6npKyCqUt/ZOH3+wC4Oroes+5uT9MGAVX9I4iIyTQzI+IBDucV88DcNPJLy+kcU5+X77y62hbUVpWIkDpMva0tK5J7cmu7KCwWWLrlMEmvfMMT//qBg78UX/Br7c7J59bXv3MWmRE9m/PhiK4qMiIeSmVGxM3ZSsp4YG4ah/NKaNEwkDn3J1LH131Pvx8TFsjMu9vz2cPX0qd1OBUOgw/WZ9Hrxf/jr//ZxtGC0rM+1zAM/pmWxc2zvmNHdj5hdf2Y/2BnJt7YGj8f/boT8VTazSTixuzlDv68KJ0d2fk0DLIy74HO1AvwMztWlWgdGcxbQzqSvi+XF5ft4Ps9x5n7XSYfpGUxrEcsw//QHJ/fzT7ll5TxxEdb+M/mQwBcGxfGywOvplFQHbN+BBGpJiozIm7KMAye/OgHvtt9jAA/b+YO7UR0fc/bjZLYLJT3hl/Dqt1HeXHZTn44kMesr3az8Pt9DPvdAuc/zV5DVm4xPl4WJiRdyYg/NHe7XW0icmlUZkTc1MvLM/h440G8vSz8/d4OJDQOMTuSy1gsFq6Na0iPK8JYti2bl5ZnsDungJeXZzjHZOUW0yTUn1fvaU+HpqEmphWR6qYyI+KGFq/dz2tf7wZg2m1tue7K2nF1Z4vFwg0JkfSNj2DJxoPMSN3JoV9KgF8vSfCnqwnx1yUJRGobrYgTcTNf/niEp/+9BYBxveMY2Cna5ETVz9vLwp8Sm/DZw9c6t80YqCIjUlupzIi4kc1ZvzBm8UYcBtyZ2ITxfeLMjmSq3x+hpEsSiNReKjMibmL/sSKGzU+juKyCP7RsSMrtbfUBLiKCyoyIWzheaGfI3HUcLbATHxnM3+/tgK+3/vqKiIDKjEiNV1JWwUPz09h7tJDG9fyZ90An6lq1dl9E5CSVGREXKLKXE/PkUmKeXEqRvfySX6fCYTDu/Y1s2P8LwXV8mP9gJxoF6yRwIiK/pzIjUkMZhsH//nc7y7Ydwc/bizfv78gVjYLMjiUiUuOozIjUUG99u5d5qzMBmHHX1XRp3sDcQCIiNZR2vIvUQP/ZfIipn/0IwKSbWjPgqiiXf88APx8yn+/v8u8jIlLVNDMjUsOs3XOMCf/cDMDQbjE8dG3seZ4hIlK7qcyI1CC7juQzfMF67BUO+rUJ5y8D4nUuGRGR81CZEakhjthKGDo3DVtJOR2a1uNvd7fHW1d9FhE5L5UZkRqgoLScB+amcfCXYpqHBfLWkE7U8fU2O5aIiFtQmRExWVmFgz8vSmf7YRthdf2Y90Bn6gf6mR1LRMRtqMyImMgwDCZ+vIVvdx3F39ebt4d0ommDALNjiYi4FZUZERPNXLGLf6UfwMsCr9/bnquj65kdSUTE7ajMiJjkg7T9/O3LXQA8d2tbrm8VbnIiERH3pJPmiZjg6505PLVkKwBjel3BoC5NTU4k1UUnJxSpepqZEalmWw/mMfrdDVQ4DG7v0JgJSS3NjiQi4tZcWmamTZtGp06dCAoKolGjRtx6663s3Lmz0hjDMJg8eTJRUVH4+/tz3XXXsW3btkpjSktLGTt2LGFhYQQGBnLLLbdw4MABV0YXcYms40UMnZtGkb2CHleE8fztV+mkeCIil8mlZWblypWMHj2a77//ntTUVMrLy0lKSqKwsNA5Zvr06cyYMYPXXnuNtLQ0IiIi6Nu3L/n5+c4x48ePZ8mSJbz//vusWrWKgoICBgwYQEVFhSvji1SpX4rsDJm7jqMFpbSKCOLv93XAz0eToyIil8ula2a++OKLSvfnzp1Lo0aNSE9P5w9/+AOGYTBz5kwmTZrE7bffDsD8+fMJDw9n8eLFjBgxgry8PN5++20WLlxInz59AFi0aBHR0dGsWLGCfv36ufJHEKkSJWUVDF+wnj0/FxIZUod5D3QmuI6v2bFERDxCtf6zMC8vD4D69esDsHfvXrKzs0lKSnKOsVqt9OzZk9WrVwOQnp5OWVlZpTFRUVEkJCQ4x5yqtLQUm81W6SZiFofDIPmfm0jLzCWojg/zHuhMREgds2OJiHiMaiszhmGQnJxMjx49SEhIACA7OxuA8PDKh6SGh4c7H8vOzsbPz4/Q0NCzjjnVtGnTCAkJcd6io6Or+scRuWAvLtvJZ1uy8fW28I/BiVwZEWR2JBERj1JtZWbMmDH88MMPvPfee6c9duoCSMMwzrso8lxjJk6cSF5envOWlZV16cFFLtP8NfsAeOnOq+nWIszkNCIinqdayszYsWP59NNP+frrr2nSpIlze0REBMBpMyw5OTnO2ZqIiAjsdju5ublnHXMqq9VKcHBwpZuImZ64oRV/bNfY7BgiIh7JpWXGMAzGjBnDxx9/zFdffUVsbGylx2NjY4mIiCA1NdW5zW63s3LlSrp16wZAYmIivr6+lcYcPnyYrVu3OseI1DS7cwqcX9/TOZqRPZubmEZExLO59Gim0aNHs3jxYj755BOCgoKcMzAhISH4+/tjsVgYP348KSkpxMXFERcXR0pKCgEBAQwaNMg5dtiwYUyYMIEGDRpQv359Hn30Udq2bes8ukmkppn+xW/nU3rqptY6l4yIiAu5tMzMnj0bgOuuu67S9rlz5zJ06FAAHn/8cYqLixk1ahS5ubl06dKF5cuXExT02yLJV155BR8fHwYOHEhxcTG9e/dm3rx5eHt7uzK+yCX5v505rNp91Hnf20tFRkTElSyGYRhmh3A1m81GSEgIeXl5Wj8jLlVe4eDGv33Lrt/tZto+pR8BfroMmisU2cuJf2YZoPdZxBNd6Oe3Tj8qUoXeS8tiV04B9QJ0QjwRkeqiMiNSRfKKy3glNQOA0b1amJxGRKT2UJkRqSKvf72b44V2WjQMZGBHnahRRKS6qMyIVIF9xwqZ910mAJP6t8bXW3+1RESqi37jilSB5z/fgb3CwbVxYfS6spHZcUREahWVGZHLtHbPMT7fmo2X5cSsjM4pIyJSvVRmRC6Dw2Hw3NIfAbirU1NaRejQfxGR6qYyI3IZlmw8yJaDedS1+pDct6XZcUREaiWVGZFLVGQv58VlJy5bMLrXFTQMspqcSESkdlKZEblEc77ZQ7athCah/jzQPcbsOCIitZbKjMglyM4r4R8r9wDw5I2tqOOr64SJiJhFZUbkEry4bCfFZRUkNgulf9tIs+OIiNRqKjMiF2nLgTw+2nAAgL8MiNeh2CIiJlOZEbkIhmHwv//dDsCt7aJoF13P3EAiIoKP2QFE3MmybdmsyzyO1ceLx29oZXacWi/Az4fM5/ubHUNETKaZGZELVFpeQcpnOwD4nz80J6qev8mJREQEVGZELtj81ZnsP15EoyArI3u2MDuOiIj8SmVG5AIcKyhl1pe7AXi035UEWrWHVkSkplCZEbkAM1fsIr+0nPjIYO7o0MTsOCIi8jv656XIeew6ks/idfuBE4die3ud/1BsLUwVEak+mpkROY+pn/1IhcMgKT6cri0amB1HREROoTIjcg4rM37m/3b+jK+3hYk3tTY7joiInIHKjMhZlFc4mLr0xAny7u8aQ2xYoMmJRETkTFRmRM7i/bQsMo4UUC/Al4evjzM7joiInIXKjMgZ2ErKeCU1A4DxveMICfA1OZGIiJyNyozIGbz+9W6OFdpp0TCQe69pZnYcERE5B5UZkVPsP1bE3FWZAEzq3xpfb/01ERGpyfRbWuQUz3/xI/YKBz2uCKPXlY3MjiMiIuehMiPyO2mZx/lsSzZeFnh6QGsslvOfIE9ERMylMiPyK4fD4H//e+JQ7Ls6NaVVRLDJiURE5EKozIj86t+bDvLDgTzqWn1I7tvS7DgiInKBVGZEgGJ7BdO/2AnAqF4taBhkNTmRiIhcKJUZEWDON3vItpXQuJ4/D3aPNTuOiIhcBJUZqfWy80p4Y+VPADx5Yyvq+HqbnEhERC6GyozUei8t30lxWQUdmtZjwFWRZscREZGL5DZl5u9//zuxsbHUqVOHxMREvv32W7MjiQfYejCPjzYcAOAvA+J1KLaIiBtyizLzwQcfMH78eCZNmsTGjRu59tprufHGG9m/f7/Z0cSNGYbBlP9uxzDgj+2iaN801OxIIiJyCdyizMyYMYNhw4bx0EMP0bp1a2bOnEl0dDSzZ882O5q4sWXbjrBu73GsPl48fkMrs+OIiMglqvFlxm63k56eTlJSUqXtSUlJrF69+ozPKS0txWazVbqJ/F5peQXTPv8RgOHXNqdxPX+TE4mIyKWq8WXm6NGjVFRUEB4eXml7eHg42dnZZ3zOtGnTCAkJcd6io6OrI6q4kYVr9rHvWBENg6z8+boWZscREZHLUOPLzEmnLsw0DOOsizUnTpxIXl6e85aVlVUdEcVNHC+087cvdwHwWNKVBFp9TE4kIiKXo8b/Fg8LC8Pb2/u0WZicnJzTZmtOslqtWK06g6uc2cwVGeSXlBMfGcwdiU3MjiMiIpepxs/M+Pn5kZiYSGpqaqXtqampdOvWzaRU4q525+Tz7toTR8E9PaA13l46FFtExN3V+JkZgOTkZAYPHkzHjh3p2rUrc+bMYf/+/YwcOdLsaOJmpi79kQqHQd/4cLq1CDM7joiIVAG3KDN33XUXx44dY8qUKRw+fJiEhAQ+++wzmjVrZnY0cSPfZPzM1zt/xsfLwsQbdSi2iIinsBiGYZgdwtVsNhshISHk5eURHBxsdhwxQXmFg5te/ZaMIwU82D2WZ26ONzuSiIicx4V+ftf4NTMiVeGD9VlkHCmgXoAv43rHmR1HRESqkMqMeDxbSRkzlmcAMK53HCEBviYnEhGRqqQyIx7v71//xLFCO80bBnLfNVpnJSLiaVRmxKNlHS/inVV7AZh0U2t8vfVHXkTE0+g3u3i05z/fgb3CQfcrGnB9q0ZmxxERERdQmRGPtT7zOEu3HMbLAk/3jz/r5S9ERMS9qcyIR3I4DP73v9sBuKtTNK0jdUi+iIinUpkRj/Tp5kNsPpBHoJ83j/RtaXYcERFxIZUZ8TjF9gpe+GIHAKN6XUGjoDomJxIREVdSmRGP8+a3ezicV0Ljev4M6xFrdhwREXExlRnxKEdsJcz+v58AeOLGVtTx9TY5kYiIuJrKjNR4RfZyYp5cSsyTSymyl59z7EvLdlJcVkGHpvW4+arIakooIiJmUpkRj7H1YB7/2nAAgKcH6FBsEZHaQmVGPIJhGDy3dDuGAbdcHUWHpqFmRxIRkWqiMiMeYfn2I3y/5zhWHy+euLGV2XFERKQaqcyI27OXO5j22Y8APHRtLI3r+ZucSEREqpPKjLi9BWsyyTxWRMMgK3++7gqz44iISDVTmRG3llto59UvdwHwaFJL6lp9TE4kIiLVTWVG3NrMFRnYSsppHRnMnxKjzY4jIiImUJkRt7U7p4BFa/cD8Jf+rfH20qHYIiK1kcqMuK2Uz36kwmHQp3U43a4IMzuOiIiYRGVG3NK3u37mqx05+HhZeOomHYotIlKbqcyI26lwGExdeuJQ7MFdm9G8YV2TE4mIiJlUZsTtfJCWxY7sfEL8fRnXO87sOCIiYjKVGXErBSXlzEjdCcC43nHUC/AzOZGIiJhNZUbcypxv93C0wE7zsEAGd21mdhwREakBVGbErSxYsw+Ap25qja+3/viKiIjKjLgZe7mDbi0a0Lt1I7OjiIhIDaEyI27FYoGn+8djsegEeSIicoLKjNR4Dofh/PqODo2Jjwo2MY2IiNQ0KjNS4y36fp/z67HX61BsERGpTGVGarRVu47y4vIM5/2GQVYT04iISE2kMiM11r5jhYxevIGK3+1mEhEROZXKjNRIBaXlDF+wnrziMto2DjE7joiI1GAqM1LjOBwGE/65iYwjBTQKsjLrnnZmRxIRkRrMZWUmMzOTYcOGERsbi7+/Py1atODZZ5/FbrdXGrd//35uvvlmAgMDCQsL4+GHHz5tzJYtW+jZsyf+/v40btyYKVOmYBja9eCpXv1qF8u2HcHP24s3BifSKLiO2ZFERKQG83HVC+/YsQOHw8E//vEPrrjiCrZu3crw4cMpLCzkpZdeAqCiooL+/fvTsGFDVq1axbFjxxgyZAiGYTBr1iwAbDYbffv2pVevXqSlpZGRkcHQoUMJDAxkwoQJroovJvliazYzV+wC4LnbEujQNJQie7nJqUREpCZzWZm54YYbuOGGG5z3mzdvzs6dO5k9e7azzCxfvpzt27eTlZVFVFQUAC+//DJDhw5l6tSpBAcH8+6771JSUsK8efOwWq0kJCSQkZHBjBkzSE5O1snTPMiObBvJ/9wEwAPdYxjYMdrcQCIi4haqdc1MXl4e9evXd95fs2YNCQkJziID0K9fP0pLS0lPT3eO6dmzJ1artdKYQ4cOkZmZecbvU1pais1mq3STmi230M7wBespslfQrUUDJt3U2uxIIiLiJqqtzPz000/MmjWLkSNHOrdlZ2cTHh5eaVxoaCh+fn5kZ2efdczJ+yfHnGratGmEhIQ4b9HR+hd+TVZe4WDMexvIOl5MdH1/Xh/UAR9dRFJERC7QRX9iTJ48GYvFcs7b+vXrKz3n0KFD3HDDDdx555089NBDlR47024iwzAqbT91zMnFv2fbxTRx4kTy8vKct6ysrIv9MaUapXy2g+92HyPAz5s37+9IaKCf2ZFERMSNXPSamTFjxnD33Xefc0xMTIzz60OHDtGrVy+6du3KnDlzKo2LiIhg7dq1lbbl5uZSVlbmnH2JiIg4bQYmJycH4LQZm5OsVmul3VJSc324Pot3vtsLwIyBV9MqQtddEhGRi3PRZSYsLIywsLALGnvw4EF69epFYmIic+fOxcur8kRQ165dmTp1KocPHyYyMhI4sSjYarWSmJjoHPPUU09ht9vx8/NzjomKiqpUmsT9bNyfy6QlWwEY1zuOGxIiTU4kIiLuyGULEw4dOsR1111HdHQ0L730Ej///DPZ2dmVZlmSkpKIj49n8ODBbNy4kS+//JJHH32U4cOHExx84l/ogwYNwmq1MnToULZu3cqSJUtISUnRkUxu7oithBEL07FXOEiKD2dcb11AUkRELo3LDs1evnw5u3fvZvfu3TRp0qTSYyfXvHh7e7N06VJGjRpF9+7d8ff3Z9CgQc5DtwFCQkJITU1l9OjRdOzYkdDQUJKTk0lOTnZVdHGxkrIKRixMJye/lJbhdZlxVzu8vFRMRUTk0liMWnAqXZvNRkhICHl5ec4ZHzGHYRg8+uEPfLThACH+vnw6pjvNGgSe8zlF9nLin1kGwPYp/Qjwc1kHFxGRGuRCP791/KtUq7nfZfLRhgN4WeD1QR3OW2RERETOR2VGqs2qXUeZ+tmPAEzqH0+PuAtbSC4iInIuKjNSLfYdK2T04g1UOAzu6NCEB7vHmB1JREQ8hMqMuFxBaTnDF6wnr7iMq6PrMfW2BB2JJiIiVUYrKcWlHA6DCf/cRMaRAhoFWZkzOJE6vt4X9RoBfj5kPt/fRQlFRMTdaWZGXOrVr3axbNsR/Ly9eGNwIuHBdcyOJCIiHkZlRlzmi63ZzFyxC4DnbkugQ9NQkxOJiIgnUpkRl9iRbSP5n5sAeKB7DAM76srlIiLiGiozUuVyC+0MX7CeInsF3Vo0YNJNrc2OJCIiHkxlRqpUeYWDMe9tIOt4MdH1/Xl9UAd8vPXHTEREXEefMlKlUj7bwXe7jxHg582b93ckNNDP7EgiIuLhVGZqmSJ7OTFPLiXmyaUU2cur9LU/XJ/FO9/tBWDGwKtpFaHrYImIiOupzEiV2LA/l0lLtgIwrnccNyREmpxIRERqC5UZuWxHbCWMXJiOvcJBUnw443rHmR1JRERqEZUZuSwlZRX8z8J0cvJLaRlelxl3tcPLS5cqEBGR6qMyI5fMMAwmLdnK5qxfCPH35c37O1LXqitkiIhI9VKZkUv2zneZfLThAF4WeH1QB5o1CDQ7koiI1EIqM3JJVu06ytSl2wGY1D+eHnFhJicSEZHaSmVGLtq+Y4WMXrwBhwF3dGjCg91jzI4kIiK1mMqMXJSC0nKGL1hPXnEZV0fXY+ptCVgsWvArIiLmUZmRC+ZwGCR/sImMIwU0CrIyZ3AidXy9zY4lIiK1nMqMXLC/fbmL5duP4OftxRuDEwkPrmN2JBEREZUZuTBfbD3M377cBcBztyXQoWmoyYlEREROUJmR89qRbSP5n5sBeKB7DAM7RpucSERE5DcqM3JOuYV2hi9YT5G9gm4tGjDpptZmRxIREalEZUbOqrzCwejFG8g6Xkx0fX9eH9QBH2/9kRERkZpFn0xyVlM/+5HVPx0jwM+bN+/vSGign9mRRERETqMycxmK7OXEPLmUmCeXUmQvNztOlfrn+izmfpcJwIyBV9MqItjcQCIiImehMiOn2bA/l6eXbAVgXO84bkiINDmRiIjI2anMSCVHbCWMXJiOvcJBUnw443rHmR1JRETknFRmxKmkrIL/WZhOTn4pLcPrMuOudnh56VIFIiJSs6nMCACGYTBpyVY2Z/1CiL8vb97fkbpWH7NjiYiInJfKjADwzneZfLThAF4WeH1QB5o1CDQ7koiIyAVRmRFW7TrK1KXbAZjUP54ecWEmJxIREblw1VJmSktLadeuHRaLhU2bNlV6bP/+/dx8880EBgYSFhbGww8/jN1urzRmy5Yt9OzZE39/fxo3bsyUKVMwDKM6onu8fccKGb14Aw4D7ujQhAe7x5gdSURE5KJUy6KIxx9/nKioKDZv3lxpe0VFBf3796dhw4asWrWKY8eOMWTIEAzDYNasWQDYbDb69u1Lr169SEtLIyMjg6FDhxIYGMiECROqI77HKiwtZ/iC9eQVl3F1dD2m3paAxaIFvyIi4l5cXmY+//xzli9fzkcffcTnn39e6bHly5ezfft2srKyiIqKAuDll19m6NChTJ06leDgYN59911KSkqYN28eVquVhIQEMjIymDFjBsnJyfrwvQxPfryFjCMFNAqyMmdwInV8vc2OJCIictFcupvpyJEjDB8+nIULFxIQEHDa42vWrCEhIcFZZAD69etHaWkp6enpzjE9e/bEarVWGnPo0CEyMzNdGd/jffljDn7eXrwxOJHw4DpmxxEREbkkLiszhmEwdOhQRo4cSceOHc84Jjs7m/Dw8ErbQkND8fPzIzs7+6xjTt4/OeZUpaWl2Gy2Sjc5s+duS6BD01CzY4iIiFyyiy4zkydPxmKxnPO2fv16Zs2ahc1mY+LEied8vTPtJjIMo9L2U8ecXPx7tl1M06ZNIyQkxHmLjo6+2B/TI+WXlPHu2n3O+4OvacrAjnpvRETEvV30mpkxY8Zw9913n3NMTEwMzz33HN9//32l3UMAHTt25N5772X+/PlERESwdu3aSo/n5uZSVlbmnH2JiIg4bQYmJycH4LQZm5MmTpxIcnKy877NZqvVhWZndj4Lv89kyYaDFNornNsf63elialERESqxkWXmbCwMMLCzn8ekldffZXnnnvOef/QoUP069ePDz74gC5dugDQtWtXpk6dyuHDh4mMPHExw+XLl2O1WklMTHSOeeqpp7Db7fj5+TnHREVFERMTc8bvbbVaTytRtU1ZhYNl27JZsGYf6/Yed25vHhbInqOFAPh46zRDIiLi/lx2NFPTpk0r3a9bty4ALVq0oEmTJgAkJSURHx/P4MGDefHFFzl+/DiPPvoow4cPJzg4GIBBgwbx17/+laFDh/LUU0+xa9cuUlJSeOaZZ3Qk0xlk55WweN1+3lu3n5/zSwHw9rLQt3U493dtxtXRIbR5drnJKUVERKqOqRff8fb2ZunSpYwaNYru3bvj7+/PoEGDeOmll5xjQkJCSE1NZfTo0XTs2JHQ0FCSk5Mr7Uaq7QzDYM2eYyxcs4/l249Q4TixpiisrpVBnaO5p0tTIkP8ASiyl5sZVUREpMpVW5mJiYk541l7mzZtyn//+99zPrdt27Z88803rormtvJLyvh4w0EWfr+P3TkFzu2dY+sz+Jpm9GsTgZ+PdiWJiIhn02WR3dDO7HwWrMlkycaDFP26oDfAz5vb2jdmcNdmtIoINjmhiIhI9VGZcRP28hMLeheu2ce6zN8W9F7RqC6Dr2nG7R0aE1TH18SEIiIi5lCZqeEO5xXz3tr9vJeWVWlBb1J8OIO7NqNr8wZaCC0iIrWaykwNZBgGa346xoI1+0j98bcFvQ2DrNzTuSmDOjclIkSXHxAREQGVmRrFVlLGx+kHWPj9Pn76udC5vXNsfe7vemJBr6/ODSMiIlKJykwNsCPbxsI1+yot6A308+a2Do0ZfE0MV0YEmZxQRESk5lKZMYm93MEX27JZdIYFvfd3bcZt7bWgV0RE5EKozFSzsy3o7dcmnMHXxHBN8/pa0CsiInIRVGaqgWEYrP7pxBl6f7+gt9GvC3rv0YJeERGRS6Yy40JnW9DbJbY+93eNIalNuBb0ioiIXCaVGRfYkW1jwZp9/PuUBb23d2jCfdc004JeERGRKqQyU0Xs5Q5W/HiIhWsyScvMdW6P+3VB761a0CsiIuISKjNVpPeMlRwrsAPg42WhX5sIBndtRpdYLegVERFxJZWZy/DSsp3Or48V2GkUZGVQlxMLesODtaBXRESkOqjMXIaoev7Or2cMvJqbr47Sgl4REZFqpk/ey/DHdlHOr29I0KUGREREzKBP38sQaNXEloiIiNlUZkRERMStqcyIiIiIW1OZEREREbemMiMiIiJuTWVGRERE3JrKjIiIiLg1lRkRERFxazpRSi0T4OdD5vP9zY4hIiJSZTQzIyIiIm5NZUZERETcmsqMiIiIuDWVGREREXFrKjMiIiLi1lRmRERExK2pzIiIiIhbU5kRERERt6YyIyIiIm5NZUZERETcmsvLzNKlS+nSpQv+/v6EhYVx++23V3p8//793HzzzQQGBhIWFsbDDz+M3W6vNGbLli307NkTf39/GjduzJQpUzAMw9XRRURExA249NpMH330EcOHDyclJYXrr78ewzDYsmWL8/GKigr69+9Pw4YNWbVqFceOHWPIkCEYhsGsWbMAsNls9O3bl169epGWlkZGRgZDhw4lMDCQCRMmuDK+iIiIuAGXlZny8nLGjRvHiy++yLBhw5zbr7zySufXy5cvZ/v27WRlZREVFQXAyy+/zNChQ5k6dSrBwcG8++67lJSUMG/ePKxWKwkJCWRkZDBjxgySk5OxWCyu+hFERETEDbhsN9OGDRs4ePAgXl5etG/fnsjISG688Ua2bdvmHLNmzRoSEhKcRQagX79+lJaWkp6e7hzTs2dPrFZrpTGHDh0iMzPTVfFFRETETbiszOzZsweAyZMn8/TTT/Pf//6X0NBQevbsyfHjxwHIzs4mPDy80vNCQ0Px8/MjOzv7rGNO3j855lSlpaXYbLZKNxEREfFMF11mJk+ejMViOedt/fr1OBwOACZNmsQdd9xBYmIic+fOxWKx8OGHHzpf70y7iQzDqLT91DEnF/+ebRfTtGnTCAkJcd6io6Mv9scUERERN3HRa2bGjBnD3Xfffc4xMTEx5OfnAxAfH+/cbrVaad68Ofv37wcgIiKCtWvXVnpubm4uZWVlztmXiIiI02ZgcnJyAE6bsTlp4sSJJCcnO+/bbDYVGhEREQ910WUmLCyMsLCw845LTEzEarWyc+dOevToAUBZWRmZmZk0a9YMgK5duzJ16lQOHz5MZGQkcGJRsNVqJTEx0Tnmqaeewm634+fn5xwTFRVFTEzMGb+31WqttMZGREREPJfL1swEBwczcuRInn32WZYvX87OnTv585//DMCdd94JQFJSEvHx8QwePJiNGzfy5Zdf8uijjzJ8+HCCg4MBGDRoEFarlaFDh7J161aWLFlCSkqKjmQSERERwMXnmXnxxRfx8fFh8ODBFBcX06VLF7766itCQ0MB8Pb2ZunSpYwaNYru3bvj7+/PoEGDeOmll5yvERISQmpqKqNHj6Zjx46EhoaSnJxcaTeSiIiI1F4WoxacStdmsxESEkJeXp5zxqcqFNnLiX9mGQDbp/QjwM+l3VBERKRWudDPb336XoYAPx8yn+9vdgwREZFaTReaFBEREbemMiMiIiJuTWVGRERE3JrKjIiIiLg1lRkRERFxayozIiIi4tZUZkRERMStqcyIiIiIW1OZEREREbemMiMiIiJuTWVGRERE3JrKjIiIiLg1lRkRERFxayozIiIi4tZUZkRERMSt+ZgdoDoYhgGAzWYzOYmIiIhcqJOf2yc/x8+mVpSZ/Px8AKKjo01OIiIiIhcrPz+fkJCQsz5uMc5XdzyAw+Hg0KFDBAUFYbFYqvS1bTYb0dHRZGVlERwcXKWvLb/R+1w99D5XD73P1UPvc/Vw5ftsGAb5+flERUXh5XX2lTG1YmbGy8uLJk2auPR7BAcH6y9LNdD7XD30PlcPvc/VQ+9z9XDV+3yuGZmTtABYRERE3JrKjIiIiLg1lZnLZLVaefbZZ7FarWZH8Wh6n6uH3ufqofe5euh9rh414X2uFQuARURExHNpZkZERETcmsqMiIiIuDWVGREREXFrKjMiIiLi1lRmLsPf//53YmNjqVOnDomJiXz77bdmR/Io06ZNo1OnTgQFBdGoUSNuvfVWdu7caXYsjzdt2jQsFgvjx483O4pHOnjwIPfddx8NGjQgICCAdu3akZ6ebnYsj1JeXs7TTz9NbGws/v7+NG/enClTpuBwOMyO5ta++eYbbr75ZqKiorBYLPz73/+u9LhhGEyePJmoqCj8/f257rrr2LZtW7VkU5m5RB988AHjx49n0qRJbNy4kWuvvZYbb7yR/fv3mx3NY6xcuZLRo0fz/fffk5qaSnl5OUlJSRQWFpodzWOlpaUxZ84crrrqKrOjeKTc3Fy6d++Or68vn3/+Odu3b+fll1+mXr16ZkfzKC+88AJvvPEGr732Gj/++CPTp0/nxRdfZNasWWZHc2uFhYVcffXVvPbaa2d8fPr06cyYMYPXXnuNtLQ0IiIi6Nu3r/P6iC5lyCXp3LmzMXLkyErbWrVqZTz55JMmJfJ8OTk5BmCsXLnS7CgeKT8/34iLizNSU1ONnj17GuPGjTM7ksd54oknjB49epgdw+P179/fePDBByttu/3224377rvPpESeBzCWLFnivO9wOIyIiAjj+eefd24rKSkxQkJCjDfeeMPleTQzcwnsdjvp6ekkJSVV2p6UlMTq1atNSuX58vLyAKhfv77JSTzT6NGj6d+/P3369DE7isf69NNP6dixI3feeSeNGjWiffv2vPnmm2bH8jg9evTgyy+/JCMjA4DNmzezatUqbrrpJpOTea69e/eSnZ1d6XPRarXSs2fPavlcrBUXmqxqR48epaKigvDw8Erbw8PDyc7ONimVZzMMg+TkZHr06EFCQoLZcTzO+++/z4YNG0hLSzM7ikfbs2cPs2fPJjk5maeeeop169bx8MMPY7Vauf/++82O5zGeeOIJ8vLyaNWqFd7e3lRUVDB16lTuueces6N5rJOffWf6XNy3b5/Lv7/KzGWwWCyV7huGcdo2qRpjxozhhx9+YNWqVWZH8ThZWVmMGzeO5cuXU6dOHbPjeDSHw0HHjh1JSUkBoH379mzbto3Zs2erzFShDz74gEWLFrF48WLatGnDpk2bGD9+PFFRUQwZMsTseB7NrM9FlZlLEBYWhre392mzMDk5Oae1Url8Y8eO5dNPP+Wbb76hSZMmZsfxOOnp6eTk5JCYmOjcVlFRwTfffMNrr71GaWkp3t7eJib0HJGRkcTHx1fa1rp1az766COTEnmmxx57jCeffJK7774bgLZt27Jv3z6mTZumMuMiERERwIkZmsjISOf26vpc1JqZS+Dn50diYiKpqamVtqemptKtWzeTUnkewzAYM2YMH3/8MV999RWxsbFmR/JIvXv3ZsuWLWzatMl569ixI/feey+bNm1SkalC3bt3P+30AhkZGTRr1sykRJ6pqKgIL6/KH2/e3t46NNuFYmNjiYiIqPS5aLfbWblyZbV8Lmpm5hIlJyczePBgOnbsSNeuXZkzZw779+9n5MiRZkfzGKNHj2bx4sV88sknBAUFOWfCQkJC8Pf3Nzmd5wgKCjptHVJgYCANGjTQ+qQq9sgjj9CtWzdSUlIYOHAg69atY86cOcyZM8fsaB7l5ptvZurUqTRt2pQ2bdqwceNGZsyYwYMPPmh2NLdWUFDA7t27nff37t3Lpk2bqF+/Pk2bNmX8+PGkpKQQFxdHXFwcKSkpBAQEMGjQINeHc/nxUh7s9ddfN5o1a2b4+fkZHTp00CHDVQw4423u3LlmR/N4OjTbdf7zn/8YCQkJhtVqNVq1amXMmTPH7Egex2azGePGjTOaNm1q1KlTx2jevLkxadIko7S01Oxobu3rr78+4+/kIUOGGIZx4vDsZ5991oiIiDCsVqvxhz/8wdiyZUu1ZLMYhmG4vjKJiIiIuIbWzIiIiIhbU5kRERERt6YyIyIiIm5NZUZERETcmsqMiIiIuDWVGREREXFrKjMiIiLi1lRmRERExK2pzIiIiIhbU5kRERERt6YyIyIiIm5NZUZERETc2v8DpOFUHvTnPWoAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.errorbar(x_data, y_data, yerr=ydataerr)\n", + "\n", + "demo_env = envs.training.EpisodeWrapper(env, \n", + " episode_length=1000, \n", + " action_repeat=1)\n", + "\n", + "render_rollout(\n", + " jax.jit(demo_env.reset),\n", + " jax.jit(demo_env.step),\n", + " jax.jit(make_inference_fn(params)),\n", + " demo_env,\n", + " n_steps=200,\n", + " camera=\"track\"\n", + ")" + ] } ], "metadata": { @@ -94,7 +1253,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.0" + "version": "3.10.13" } }, "nbformat": 4,