Discussion:
[sage-devel] GSOC 2016-Regression test framework
Jayamine A.
2016-02-26 02:50:18 UTC
Permalink
Hi,

I would like to ask your opinion about the question in below.

testing can be divied into
1) Unit testing
2) Integrated testing

When we are dealing with the server side,we can use unit testing for each
function/method.

But when we are dealing with clients(users) we need both.(My opinion)

I would like to ask about your opinions and please share your thoughts with
me.

Thanks
Jayamine
Hello Jayamine,
I just answer about "Regression test framework" that I did propose. It
is pure Python project and might fit with your skills. It is actually
- users should be able to propose small snippet of programs to be
included in the base of tests
- developer should set up and maintain servers that run the regression
tests
Ideally, the GSoC project could even include a web-app to submit python
code and see performance evolution.
Best,
Vincent
Hi ,
I'm an third year computer engineering student at university of
Peradeniya.
My major subjects are Discrete mathematics, Software engineering and
Networking.
Actually I'm new to Sage .. I played with sage to get a real feel of
sage.
Graphs, algorithms!!
It's awesome..
Sincerely I have a good knowledge about python but it is not very good.
But i'm quite familiar with object oriented programming(java, C# ,
objective c)
So i think improving my knowledge is not a difficult task.I'm working on
it.
I feel Sage is really important to students specially students in
developing countries like us.
Therefore I want to contribute sage even after gsoc.
I think gsoc is a good opportunity to start with Sage.
Specially I like Regression test framework project and Hyperplane
arragement projects.
*Hyper-plane arrangement*
I would like to know more about the current situation of Hyperplane
arragement in sage and
how about The Orlik–Solomon algebra and The intersection semilattice
and
the matroid with hyperplane arragement?
If there are any kind of references please let me know :-)
*Regression test framework*
Look like my next question is a foolish question.Sorry for it.
Having a good testing framework is important . But
Who will use this test framework? Developers or users?
If there are any kind of references which will be help please let me
know.
Thank you.
Jayamine :-)
--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+***@googlegroups.com.
To post to this group, send email to sage-***@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Clemens Heuberger
2016-02-26 07:19:18 UTC
Permalink
Post by Jayamine A.
But when we are dealing with clients(users) we need both.(My opinion)
That was basically my opinion in the thread
https://groups.google.com/d/topic/sage-devel/ojHqBy7oUVU/discussion

Clemens
--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+***@googlegroups.com.
To post to this group, send email to sage-***@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Jayamine A.
2016-02-26 17:05:25 UTC
Permalink
To get an idea i read the whole conversation

What i gained is


On Friday, February 26, 2016 at 12:49:32 PM UTC+5:30, Clemens Heuberger
Post by Clemens Heuberger
Post by Jayamine A.
But when we are dealing with clients(users) we need both.(My opinion)
That was basically my opinion in the thread
https://groups.google.com/d/topic/sage-devel/ojHqBy7oUVU/discussion
Clemens
--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+***@googlegroups.com.
To post to this group, send email to sage-***@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Jayamine A.
2016-02-26 18:10:06 UTC
Permalink
I read the whole conversation to get an idea

It is about server side(we can call it developers side).
1) check the accuracy of each function/method
2) check the time for each function/method

Is there other opinions about client side testing?


On Friday, February 26, 2016 at 12:49:32 PM UTC+5:30, Clemens Heuberger
Post by Clemens Heuberger
Post by Jayamine A.
But when we are dealing with clients(users) we need both.(My opinion)
That was basically my opinion in the thread
https://groups.google.com/d/topic/sage-devel/ojHqBy7oUVU/discussion
Clemens
--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+***@googlegroups.com.
To post to this group, send email to sage-***@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Erik Bray
2016-02-29 09:41:25 UTC
Permalink
On Fri, Feb 26, 2016 at 7:10 PM, Jayamine A.
Post by Jayamine A.
I read the whole conversation to get an idea
It is about server side(we can call it developers side).
1) check the accuracy of each function/method
2) check the time for each function/method
Is there other opinions about client side testing?
Hi,

I apologize if I don't fully follow the context to this thread, but I
saw the mention of performance testing, in particular in the context
of regression tests, and I thought I would bring up a project Mike
Droettboom developed called airspeed velocity, which is a nice simple
package for tracking performance of tests over time:

http://asv.readthedocs.org/en/latest/

One writes very simple tests (often just one or two liners) and asv
records the time taken to run that test and/or the memory usage (and
in principle other statistics could be plugged in). Obviously the
benchmarks are very machine-dependent too, so each benchmark tries to
include metadata about the platform. So the best way to use it is
have some dedicated machines, whose hardware doesn't change that
often, run the benchmark suite continuously. It outputs the
benchmarks to a JSON file that can be kept under version control and
used to update the generated benchmark history. You can see a sample
of Astropy's benchmarks here:

http://droettboom.com/astropy-benchmark/

It has already proven itself several times to be very useful at
catching noticeable performance regressions. Might be worth a look
(disclaimer I worked on this project a bit too, but have no financial
incentive for promoting it etc. etc. :)

Erik
Post by Jayamine A.
On Friday, February 26, 2016 at 12:49:32 PM UTC+5:30, Clemens Heuberger
Post by Clemens Heuberger
Post by Jayamine A.
But when we are dealing with clients(users) we need both.(My opinion)
That was basically my opinion in the thread
https://groups.google.com/d/topic/sage-devel/ojHqBy7oUVU/discussion
Clemens
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+***@googlegroups.com.
To post to this group, send email to sage-***@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Jeroen Demeyer
2016-02-29 13:08:23 UTC
Permalink
Concerning benchmarking, let me state that timing the tests is the easy
part. The hard part is doing something useful with the results.

For SageMath, the "doing something useful" part should be to make this
part of the continuous integration tests such that patches which
significantly slow down tests are flagged. Ideally, with very little
false positives (I think that the test for startup time in the patchbot
has too many false positives).
--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+***@googlegroups.com.
To post to this group, send email to sage-***@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Volker Braun
2016-02-29 15:02:22 UTC
Permalink
Post by Jeroen Demeyer
significantly slow down tests are flagged. Ideally, with very little
false positives (I think that the test for startup time in the patchbot
has too many false positives).
I'm pretty sure that they are actual slowdowns, not statistical noise. The
problem is that we actually have a quite good temporal resolution so some
weird effects can be measured; E.g. some builds are slower or faster
depending on disk layout of files.
--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+***@googlegroups.com.
To post to this group, send email to sage-***@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Jayamine A.
2016-02-29 17:17:34 UTC
Permalink
Post by Erik Bray
Hi,
I apologize if I don't fully follow the context to this thread, but I
saw the mention of performance testing, in particular in the context
of regression tests, and I thought I would bring up a project Mike
Droettboom developed called airspeed velocity, which is a nice simple
http://asv.readthedocs.org/en/latest/
One writes very simple tests (often just one or two liners) and asv
records the time taken to run that test and/or the memory usage (and
in principle other statistics could be plugged in). Obviously the
benchmarks are very machine-dependent too, so each benchmark tries to
include metadata about the platform. So the best way to use it is
have some dedicated machines, whose hardware doesn't change that
often, run the benchmark suite continuously. It outputs the
benchmarks to a JSON file that can be kept under version control and
used to update the generated benchmark history. You can see a sample
http://droettboom.com/astropy-benchmark/
It has already proven itself several times to be very useful at
catching noticeable performance regressions. Might be worth a look
(disclaimer I worked on this project a bit too, but have no financial
incentive for promoting it etc. etc. :)
Erik
Thanks for your help.Actually I did not think about that side. The
performance that can track using ASV mostly depends on hardware
and other externally related things. Actually i thought it is not related
to regression testing. thanks a lot.As you mentioned there are some
questions about the hardware part.
(How to match it to this project. first I have to study). :-)
Post by Erik Bray
For SageMath, the "doing something useful" part should be to make this
part of the continuous integration tests such that patches which
significantly slow down tests are flagged. Ideally, with very little
false positives (I think that the test for startup time in the patchbot
has too many false positives).
In that case i did not get the "significantly slow down" part? But i think
that you meant about compare that time that the funtion/unit took before
patching and after patching?
In that case we need a database and compare with the new time duration or
something similar to it. :-)
--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+***@googlegroups.com.
To post to this group, send email to sage-***@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Loading...