Ask! v0.3 Report

12 weeks ago, Patract submitted #101 treasury proposal of Ask! for proposed features including implementation designs, principles and process. In v0.3 we are implementing the following features:

Goals for ask! v0.3: providing cli management tools and better ask! coding experience and execution performance
new project management tool: ask-cli
better execution performance
system parameter types in custom env
unit test and documentation

The code of implementation is in our git repo Ask!. For contract examples written in Ask!, please refer examples. You can check documentation on docs.patract.io. Please review it on branch v0.3-review as it will be merged into master later.

Design and implementations

Based upon Ask! v0.2, we introduced ask-cli as the the command line tool to manage contract development. We optimized the Ask! execution performance. Additionally, we also provided related documentations.

New Project Management Command-line Tool ask-cli

ask-cli is the Ask! command-line tool for managing the contract compilation lifecycling.
It provides init and compile functions:

• ask-cli init:
  init is used for initializing Ask! contract projects. init read from Dependencies for latest project and updates corresponding NPM packages and the creates local directory structured like:

.
├── build
├── contracts
├── node_modules
└── package.json

Inside the dir, contracts contains the source code of contract. build is generated once compilation is done and contains .wasm and metadata.json. If the compilation is done in debug mode, build may contain other files as well.

• ask-cli compile [--release|--debug] contracts/Hello.ts:
  compile is used to compile targeting file of contract source code and generate .wasm and metadata.json in build directory.
  compile contains two compiling modes: release and debug. While --release is the defaul option to compile under highest level of optimization and compression. --debug is the debug mode which will generate other files created in compilation.

For detailed usages, please refer related chapters in QuickStart.

Performance optimization

2.1 Merging the functions of @storage into @contract, simplifying the process of state variable definition.

Before v0.3, state variables are defined seperately in @storage, which does't support contract inheritence well. Therefore in v0.3, we put stated varaible definitions directly in @contract and removed @storage.

In v0.2, before we define @contract class, we need to define @storage class, then define storage property in @contract. However, storage is a property in @contract class. If we want to add storage property in child class during inheritence, we would redefine the @storage class. in v0.3, for inheritence, if we want to add a variable we can simply add it in subclass.

eg. in v0.2:

For inheritence with extra properties in @stroage in v0.2:

@storage
class ERC20StoragePausable extends ERC20Storage{
    is_pausable: bool;
}

@contract
export class ERC20Pausable {
  private storage: ERC20StoragePausable;}

In v0.3

@contract
export class ERC20Pausable extends ERC20 {
  @state is_pausable: bool = false;}

At the same time, we introduced @state decorator to mark the specific member variable as state variable while the ones not decorated are class variables. In v0.2, all variables are default as blockchain state variables. Since we moved @storage into @contract class for better inheritence, we now have to sperate blockchain state variables and normal class properties by having @state decorator.

@storage
class ERC20Storage {
  balances: SpreadStorableMap<AccountId, UInt128>;
  allowances: SpreadStorableMap<AccountId, SpreadStorableMap<AccountId, UInt128>>;

    totalSupply: u128;
    name: string;
    symbol: string;
    decimal: u8;
}

@contract
export class ERC20 {
  private storage: ERC20Storage;}

In v0.3, we now define storage directly inside the @contract class with @state:

@contract
export class ERC20 {
  @state balances: SpreadStorableMap<AccountId, UInt128> = new SpreadStorableMap<AccountId, UInt128>();
  @state allowances: SpreadStorableMap<AccountId, SpreadStorableMap<AccountId, UInt128>> = new SpreadStorableMap<AccountId, SpreadStorableMap<AccountId, UInt128>>();

  @state totalSupply: u128 = u128.Zero;
  @state name_: string = "";
  @state symbol_: string = ""
  @state decimal_: u8 = 0;}

2.2 Optimize key generation logic for storing state variables. Use sequential hash data instead of hash data generated by dynamic hash(string)

While a contract is inherited, all @state decorated variables are sorted by their definition order and baseclass/subclass relationship. And the order sequence number will serve as the id of state changes in storages.

• First, the compiler will first locate the entry of the contract

      this.program.elementsByName.forEach((element, _) => {
      		let contractNum = 0;
            if (ElementUtil.isTopContractClass(element)) {
                contractNum++;
                this.contract = new ContractInterpreter(<ClassPrototype>element);
            }
        });

Then iterate through the base classes to push the objects to be stored into stack:

 private resolveBaseClass(classPrototype: ClassPrototype): void {
  if (classPrototype.basePrototype) {
      let basePrototype = classPrototype.basePrototype;
      basePrototype.instanceMembers &&
          basePrototype.instanceMembers.forEach((instance, _) => {
              if (ElementUtil.isField(instance)) {
                  let fieldDef = new FieldDef(<FieldPrototype>instance);
                  if (!fieldDef.decorators.ignore) {
                      this.storeFields.push(fieldDef);
                  }
              }
          });
      this.resolveBaseClass(basePrototype);
  }
}

When a new @contract class inherits from parent @contract as a child class, the new @state properties defined in child class will also be sequence.

@contract
export class ERC20 {
  @state balances: SpreadStorableMap<AccountId, UInt128> = new SpreadStorableMap<AccountId, UInt128>();
  @state allowances: SpreadStorableMap<AccountId, SpreadStorableMap<AccountId, UInt128>> = new SpreadStorableMap<AccountId, SpreadStorableMap<AccountId, UInt128>>();

  @state totalSupply: u128 = u128.Zero;
  @state name_: string = "";
  @state symbol_: string = ""
  @state decimal_: u8 = 0;
}

Adding a new class property with @state

class MyToken extends ERC20 {
  @state is_paused:bool = false;}

In the compiled metadata.json, we can the new @state is_paused is sequenced correctly under inheritence:

        {
          "name": "symbol_",
          "layout": {
            "key": "0x0000000000000000000000000000000000000000000000000000000000000005",
            "ty": 1
          }
        },
        {
          "name": "decimal_",
          "layout": {
            "key": "0x0000000000000000000000000000000000000000000000000000000000000006",
            "ty": 2
          }
        },
        {
          "name": "is_pause",
          "layout": {
            "key": "0x0000000000000000000000000000000000000000000000000000000000000007",
            "ty": 5}}

2.3 During a contract message call, when a state variable is mutated multiple times, reduce the number of seal_set_stroage calls

@state introduces lazy option as: @state({"lazy": false})
While lazy is true, that means while a state variable gets changed multiple times in a contract call, only the last change will be synced to blockchain. The default value of lazy is true. While lazy is false, then every change made to the state variable will be synced to blockchain.

Basic principle of implmentation:
For every state varible with lazy set as true, the setter function generated by compiler will only updates the value changed in memory; Meanwhile, compiler also creates a __commit__ function. If the state variables within this function ever gets changed before the contract call is done, the updated values will be synced to blockchain.

Using object type bool as the example, when lazy is set to false. The setter method generated follows:

  set vbool(newvalue: bool) {
      this._vbool = new Bool(newvalue);
      const st = new Storage(new Hash("0x0000000000000000000000000000000000000000000000000000000000000001"));
      st.store<Bool>(this._vbool!);
  }
  

When lazy is set to true. The setter and __commit__ functions generated are:

 set vbool(v: bool) {
     this._vbool = new _lang.Bool(v);
 }
 __commit_storage__(): void { 
 if (this._vbool !== null) {
     const st = new _lang.Storage(new _lang.Hash([0x0000000000000000000000000000000000000000000000000000000000000001]));
     st.store<_lang.Bool>(this._vbool!);
      }
 }

To verify, write a simple contract as follows. Because we do not state @state({"lazy": false}) on
@state flag: bool. Even we are modifying it multiple times in flip(). It will call seal_set_storaqe once. You can monitor it in Europa logs that seal_set_storaqe only gets called once.

@contract
class Flipper {
    @state flag: bool;

    constructor() {
    }

    @constructor
    default(initFlag: bool): void {
        this.flag = initFlag;
    }

    @message
    flip(): void {
        const v = this.flag;
        this.flag = !v;
        this.flag = !v;
        this.flag = !v;
    }

    @message({"mutates": false})
    get(): bool {
        return this.flag;
    }
}

It the log printed by Europa,

1: NestedRuntime {
    ext_result: [success] ExecReturnValue { flags: 0, data:  },
    caller: d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d (5GrwvaEF...),
    self_account: 9b1b5687f0e868a1ab3b5536efbe3dfe14aea17570d605c01ef868d5d53e51c0 (5Fa5QB7h...),
    selector: 0x633aa551,
    args: None,
    value: 0,
    gas_limit: 99827000000,
    gas_left: 253627,
    env_trace: [
        seal_input(Some(0x633aa551)),
        seal_get_storage((Some(0x0000000000000000000000000000000000000000000000000000000000000001), Some(0x00))),
        seal_set_storage((Some(0x0000000000000000000000000000000000000000000000000000000000000001), Some(0x01))),
    ],
    sandbox_result_ok: Value(
        I32(
            0,
        ),
    ),
    nests: [],
}

The selector is defined in metadata.json

      {
        "mutates": true,
        "payable": false,
        "args": [],
        "returnType": null,
        "docs": [
          ""
        ],
        "name": [
          "flip"
        ],
        "selector": "0x633aa551"
      },

2.4 Define the export format of Map and Array in metadata.json

• The export format of Array have two parts: type definition and store definition. In ask!, by default, the array is mutable length array. The default strucutre definition is SequenceDef that defines array as sequence and sepcify the object type in array. It also defines storage modes as pack/spread. In addition, for type array, it can pre-allocate some space by default. The type is Arraydef whith specification of capacity for pre-allocated space. len is set to 0 by default meaning no fixed length is specified.

export interface Type extends ToMetadata {
    typeKind(): TypeKind;

    toMetadata(): ITypeDef;
}

export class SequenceDef implements Type {
    constructor(public readonly type: number) {}

    typeKind(): TypeKind {
        return TypeKind.Sequence;
    }

    toMetadata(): ISequenceDef {
        return {
            def: {
                sequence: {
                    type: this.type,
                },
            },
        };
    }
}

export class ArrayDef implements Type {
    constructor(public readonly len: number, public readonly type: number) {}

    typeKind(): TypeKind {
        return TypeKind.Array;
    }

    toMetadata(): IArrayDef {
        return {
            def: {
                array: {
                    len: this.len,
                    type: this.type,
                },
            },
        };
    }
}

SequenceDef generates the following format：

{
      "def": {
        "sequence": {
          "type": 4
        }
      }
 }

ArrayDef generates the following format：

 {
      "def": {
        "array": {
          "len": 32,
          "type": 2
        }
      }
    }

Their storage structures looks like

{
  "name": "ages",
  "layout": {
    "struct": {
      "fields": [
        {
          "name": "len",
          "layout": {
            "key": "0x0000000000000000000000000000000000000000000000000000000000000002",
            "ty": 3
          }
        },
        {
          "name": "elems",
          "layout": {
            "offset": "0x0000000000000000000000000000000000000000000000000000000000000002",
            "len": 0,
            "cellsPerElem": 1,
            "layout": {
              "key": "0x0000000000000000000000000000000000000000000000000000000000000002",
              "ty": 3
            },
            "storemode": "spread"
          }
        }
      ]
    }
  }
}

The differences between SequenceDef without capacity limit and ArrayDef with capacity limit: len of SequenceDef is 0 while len of ArrayDef is not 0

{
  "name": "elems",
  "layout": {
    "offset": "0x0000000000000000000000000000000000000000000000000000000000000002",
    "len": 0,
    "cellsPerElem": 1,
    "layout": {
      "key": "0x0000000000000000000000000000000000000000000000000000000000000002",
      "ty": 3
    },
    "storemode": "spread"
}

To verify, write a simple contract by importing PackedStorableArray:

import { PackedStorableArray, UInt128} from "ask-lang";

@contract
class Flipper {
    @state flag: bool;

    @state
    @packed({ "capacity": 128 })
    packeArr: PackedStorableArray<UInt128> = new PackedStorableArray<UInt128>();

    @state
    aArr: PackedStorableArray<UInt128> = new PackedStorableArray<UInt128>();

    constructor() {
    }

    @constructor
    default(initFlag: bool): void {
        this.flag = initFlag;
    }

    @message
    flip(): void {
        const v = this.flag;
        this.flag = !v;
    }

    @message({"mutates": false})
    get(): bool {
        return this.flag;
    }
}

In the compiled metadata.json, we can see SequenceDef and ArrayDef have different len:

    {
      "def": {
        "array": {
          "len": 128,
          "type": 2
        }
      }
    },
    {
      "def": {
        "sequence": {
          "type": 2
        }
      }
    }

If we compile in --debug mode, in the pre-compiled code generated:

 get packeArr(): PackedStorableArray<UInt128> {
     if (this._packeArr === null) {
       this._packeArr = new _lang.PackedStorableArray<UInt128>(new _lang.Hash([0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x02]), true, 128);
     }
     return this._packeArr!;
  }
  
  get aArr(): PackedStorableArray<UInt128> {
     if (this._aArr === null) {
       this._aArr = new _lang.PackedStorableArray<UInt128>(new _lang.Hash([0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x03]), true, 0);
     }
     return this._aArr!;
  }

• For map storage type, it requires the key data type and value data type of object to be stored. It also needs to specify the storage modes as pack or spread and entry key value of the object to be stored.

export class CompositeDef implements Type {
    constructor(public readonly fields: Array<Field>) {}

    typeKind(): TypeKind {
        return TypeKind.Composite;
    }
    toMetadata(): ICompositeDef {
        return {
            def: {
                composite: {
                    fields: this.fields.map((f) => f.toMetadata()),
                },
            },
        };
    }
}

The generated format for storage instance looks like：

{
    "def": {
      "composite": {
        "fields": [
          {
            "name": "key_index",
            "type": 2
          },
          {
            "name": "value",
            "type": 3
          }
        ]
      }
    }
  }，
  {
    "def": {
      "primitive": "u8"
    }
  }，
      {
    "def": {
      "primitive": "str"
    }
  }

The storage structure looks like

{
  "name": "allowances",
  "layout": {
    "struct": {
      "fields": [
        {
          "name": "key",
          "layout": {
            "offset": "0x0000000000000000000000000000000000000000000000000000000000000002",
            "strategy": {
              "hasher": "Blake2x256",
              "prefix": "0x0000000000000000000000000000000000000000000000000000000000000002",
              "postfix": ""
            },
            "layout": {
              "key": "0x0000000000000000000000000000000000000000000000000000000000000002",
              "ty": 3
            },
            "storemode": "spread"
          }
        },
        {
          "name": "values",
          "layout": {
            "offset": "0x0000000000000000000000000000000000000000000000000000000000000002",
            "strategy": {
              "hasher": "Blake2x256",
              "prefix": "0x0000000000000000000000000000000000000000000000000000000000000002",
              "postfix": ""
            },
            "layout": {
              "key": "0x0000000000000000000000000000000000000000000000000000000000000002",
              "ty": 6
            },
            "storemode": "spread"
          }
        }
      ]
    }
  }
}

2.5

2.5 Use JSON format instead of bare() annotations.

This change improves code readibility and makes easier to for compiler to interprete.
eg. @message(selector = '0x00001111') is now @message({"selector": "0x00001111"})

• First parse the anotated parts as json object. eg. “{"selector": "0x00001111"}”

export class DecoratorNodeDef {
    jsonObj: any;
    constructor(public decorator: DecoratorNode) {
        this.jsonObj = this.parseToJson(decorator);
    }
}

For specific decorator, compiler will parse with specific decorator class and run class specific checks.

export class MessageDecoratorNodeDef extends DecoratorNodeDef {
    constructor(decorator: DecoratorNode, public payable = false,
        public mutates = true, public selector = "") {
        super(decorator);
        this.payable = this.getIfAbsent("payable", false, "boolean");
        this.mutates = this.getIfAbsent('mutates', true, "boolean");
        if (this.hasProperty('selector')) {
            this.selector = this.getProperty('selector');
            DecoratorUtil.checkSelector(decorator, this.selector);
        }
        if (this.payable && !this.mutates) {
            throw new Error(`Decorator: ${decorator.name.range.toString()} arguments mutates and payable can only exist one. Trace: ${RangeUtil.location(decorator.range)} `);
        }
    }
}

2.6 Enhance Event syntax

In v0.2, we introduced @event decorator to emit Event. However, in v0.2, Event can't be inheriteted and Event will emit once it gets instantiated, which isn't very intuitive for programmers. Therefore, we made the following optimization in v0.3:

• To implement @event, developer has to either inherit from __lang.Event or from another Event
• To emit an Event, developers has to call .emit()

The new Event usages looks like:

@event
class EventA extends __lang.Event {

  @topic topicA: u8;
  name: string;

  constructor(t: u8, n: string) {
      super();
      this.topicA = t;
      this.name = n;
  }
 }

@event
class EventB extends EventA {
  @topic topicB: u8;
  gender: string;
  constructor(t: u8, g: string) {
      super(t, g);
      this.topicB = t;
      this.gender = g;
  }
}

@contract
export class EventEmitter {

    count: i8;

    constructor() {
    }

    @message
    triggeEventA(): void {
        let eventA = new EventA(100, "Elon");
        eventA.emit();
    }

    @message
    triggeEventB(): void {
         let eventB = new EventB(<u8>300, "M");
         eventB.emit();
     }
}

Currently, due to the lack of contract standards in pallet-contract, polkadot.js/app is not able to parse event correctly. Therefore, the event emission can not be verified from the polkadot.js/app frontend or europa logs.

note: currently, Event class does not support inheritence.

2.7 Enhance decoroter syntax and parameter checks

In v0.2, compiler will only report wrong decorator. Eg.@massage, compiler will only report contract doesn't support @massage decorator. (Spelling error)

    @massage({"mutates": false})
    get(): bool {
        return this.flag;
    }

With the enhanced checks, compiler will utilize string match algorithm to predict that the user intends to input @message as the decorator and provides hints:

Unsupported contract decorator @massage, do you mean '@message'? Check source text: @massage({"mutates": false}) in path:examples/flipper/flipper.ts lineAt: 24 columnAt: 5 range: (346 374).

It will also check if @message is marked as public function with the following error message:

Decorator[@message] should mark on public method(Method: get isn't public method). Check source text: @message({"mutates": false})
    @message({"mutates": false})
    private get(): bool {
        return this.flag;
    } in path:examples/flipper/flipper.ts lineAt: 24 columnAt: 5 range: (346 432)..

The checker will also check unsupported keywords in the decorator:

    @message({"mutates": false, "superInherit": true})
    get(): bool {
        return this.flag;
    }

It will report the error:

FAILURE The parameter: superInherit isn't pre-defined in decorator @message, do you mean selector? Check source text: @message({"mutates": false, "superInherit": true}) in path:examples/flipper/flipper.ts lineAt: 25 columnAt: 5 range: (347 397)..

2.8 Optimize the size of the generated wasm file.

In v0.3, by default,ask-cli will compile in --release mode so the compiler will use option -o3z to optimize and compress the wasm file generated. In addtion, in the Framework, we reduces the resources comsumed by string to shrink the codes of Framework

2.9 Upgrade the seal_xxx method in pallet-contrac

The method seal_xxx used in contract is now updated to latest seal0 of Europa

Provide system parameter types in custom env .

• By default, AccountId,Hash,Balance,BlockNumber are implemented as Array(32), Array(32), UInt128, UInt32.
• You can now customize it in assembly/env/CustomTypes.ts as long as the correct Codec is implementd

Unit Testing and Documentation.

• Provide exmples to test the features in our Ask! Framework in examples/
• Provide tests/ to test the compiler. In ts-package, we provide ts-packages/contract-metadata/src/ and ts-packages/transform/src/__tests__/ for tests we used.
  To run the unit-test:

cd ts-packages
yarn jest

You should see the following log, showing all unit tests are passing:

yarn run v1.22.11
$ /home/bonan/repos/ask/ask-compiler/node_modules/.bin/jest
 PASS  ts-packages/contract-metadata/dist/index.spec.js
 PASS  ts-packages/transform/src/__tests__/generator.test.ts
 PASS  ts-packages/contract-metadata/src/index.spec.ts
 PASS  ts-packages/transform/src/__tests__/types.test.ts
 PASS  ts-packages/transform/src/__tests__/decorator.test.ts

For documentations, please refer QuickStart.

Start using Ask! v0.3

Ask! v0.3 is now released, please refer QuickStart to quick start it.

For detailed usages of components in Ask!, please refer API Usages.

Quick start

Now, let's use pl-ask-cli to compose ask! smart contracts.

1. init a new directory: mkdir erc20
2. cd erc20
3. init an npm project: npm init -y
4. install pl-ask-cli: npm i pl-ask-cli
5. init project: npx pl-ask-cli init
6. copy index.ts in example/erc20 , ERC20.ts to erc20/contracts/.
7. compile: npx pl-ask-cli compile contracts/index.ts

Once compiled successfully, we can deploy and call the contract.

Use ERC20 base Contract

ERC20.ts is the base class that implments ERC20 standard with reusable ERC20 interfaces such as transfer, approve etc. It defines the storages for contract as well as Event of Transfer and Approval.

In Ask! v0.3, we have reimplemented ERC20 with new coding conventions. So the new contract can still be written like:

import { Account, u128 } from "ask-lang";
import {ERC20} from "./ERC20";

@contract
@doc({"desc": "MyToken conract that extended erc20 contract"})
class MyToken extends ERC20 {

  constructor() {
    super();
  }

  @constructor
  default(name: string = "", symbol: string = ""): void {
    super.default(name, symbol);
  }


  @message
  @doc({"desc": "Mint a token"})
  mint(to: Account, amount: u128): void {
    this._mint(to, amount);
  }

  @message
  @doc({"desc": "burn the token"})
  burn(from: Account, amount: u128): void {
    this._burn(from, amount);
  }
}

Compile

To compile the contract:

$ npx ask-cli compile contracts/index.ts

After successfuly compilation, wasm and metadata.json will be generated under examples/erc20/build/.

Deployment and contract calls

We use Europa(v3.0.0 branch) sandbox to deploy and test contracts with polkadot-js(master branch, commit-id 11276477a0523348c7b143db566622aa32833296) as the frontend
Test:

1. Follow the instructions of Europa and plokadot-js to start node and services.

2. In polkadot-js contract tab, upload build/metadata.json and build/target.wasm.

3. Instantiate the uploaded contract and call default to issue tokens.

4. call mint, transfer, approve, burn to operate this ERC20 contract.

Now, with ask-cli and new ask! contract features, we succesfully issued ERC20 tokens.

Implemented features of Ask! v0.3:

• release of new Ask v0.3 and npm of ask-cli .
• Implementing contracts in /examples with new base contracts .
• Complete contract development tutorial
• Complete API documentations.