Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

[MXNET-101] Support float16 in LeakyReLU operator#10169

Merged
piiswrong merged 6 commits intoapache:masterfrom
haojin2:fp16_leaky_relu
Mar 22, 2018
Merged

[MXNET-101] Support float16 in LeakyReLU operator#10169
piiswrong merged 6 commits intoapache:masterfrom
haojin2:fp16_leaky_relu

Conversation

@haojin2
Copy link
Contributor

@haojin2 haojin2 commented Mar 20, 2018 •
edited
Loading

Description

Add support for any datatype for Leaky ReLU operator, mentioned in issue #2302.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

  • Change LeakyReLU operator from only supporting real_t to supporting any datatype
  • Add test coverage for "elu" activation type of LeakyReLU
  • Add test coverage for "leaky" activation type of LeakyReLU
  • Add test coverage for "prelu" activation type of LeakyReLU

Comments

The test for rrelu is harder to write as rrelu contains random components in forward pass, will figure out a way to write a test for it later.

@haojin2 haojin2 requested a review from cjolivier01 as a code owner March 20, 2018 06:32
@haojin2 haojin2 force-pushed the fp16_leaky_relu branch 2 times, most recently from 0fcaa7d to 512feea Compare March 20, 2018 22:16
marcoabreu
marcoabreu reviewed Mar 20, 2018
View reviewed changes
src/operator/leaky_relu-inl.h

template<typename xpu>
Operator* CreateOp(LeakyReLUParam type);
Operator* CreateOp(LeakyReLUParam type, int dtype);
Copy link
Contributor

@marcoabreu marcoabreu Mar 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I'm not familiar with the cpp code, but is there no typedef for dtype - int is kind of arbitrary.

Copy link
Contributor Author

@haojin2 haojin2 Mar 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The types are represented by integers, please see leaky_relu.cc:43 for how this function is being called. Also you can refer to https://github.com/dmlc/mshadow/blob/b3771de20ed36f90ba7b8436ae4b79ea298a687a/mshadow/base.h#L288 for more info about how each integer map to a type. This change is written in a similar way as #2322 #3011 #10078 #10125 .

Copy link
Contributor

@marcoabreu marcoabreu Mar 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks for the explanation :)

Copy link
Contributor Author

@haojin2 haojin2 Mar 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My pleasure! 😄

@haojin2 haojin2 force-pushed the fp16_leaky_relu branch 5 times, most recently from fb0c6c2 to 5390b43 Compare March 22, 2018 00:12
@piiswrong piiswrong merged commit 10d7458 into apache:master Mar 22, 2018
@haojin2 haojin2 deleted the fp16_leaky_relu branch March 26, 2018 21:30
ashokei pushed a commit to ashokei/incubator-mxnet that referenced this pull request Mar 27, 2018
@haojin2 @ashokei
[MXNET-101] Support float16 in LeakyReLU operator (apache#10169)
5ee08c7 
* support for any datatype in leaky ReLU

* test for LeakyReLU operators

* make lint

* clean up unnecessary prints

* fix for amalgamation build failure

* add InferType for Leaky ReLU and slight modification to the tests
jinhuang415 pushed a commit to jinhuang415/incubator-mxnet that referenced this pull request Mar 30, 2018
@haojin2
[MXNET-101] Support float16 in LeakyReLU operator (apache#10169)
ac400f9 
* support for any datatype in leaky ReLU

* test for LeakyReLU operators

* make lint

* clean up unnecessary prints

* fix for amalgamation build failure

* add InferType for Leaky ReLU and slight modification to the tests
rahul003 pushed a commit to rahul003/mxnet that referenced this pull request Jun 4, 2018
@haojin2 @piiswrong
[MXNET-101] Support float16 in LeakyReLU operator (apache#10169)
9f85a42 
* support for any datatype in leaky ReLU

* test for LeakyReLU operators

* make lint

* clean up unnecessary prints

* fix for amalgamation build failure

* add InferType for Leaky ReLU and slight modification to the tests
zheng-da pushed a commit to zheng-da/incubator-mxnet that referenced this pull request Jun 28, 2018
@haojin2 @zheng-da
[MXNET-101] Support float16 in LeakyReLU operator (apache#10169)
8623f58 
* support for any datatype in leaky ReLU

* test for LeakyReLU operators

* make lint

* clean up unnecessary prints

* fix for amalgamation build failure

* add InferType for Leaky ReLU and slight modification to the tests
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@cjolivier01 cjolivier01 Awaiting requested review from cjolivier01

1 more reviewer

@marcoabreu marcoabreu marcoabreu left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@haojin2 @marcoabreu @piiswrong