oDesk Tests - Online Skill Assessment Tests

Although I passed the Access 2003 test I must say it stinks. The test doesn't test your knowledge of developing in Access. It has to many questions on small techinical things that when you need to use them you will look in help or online and find out what to do. I don't even develop in 2003 and I still passed (I use 2000 or XP).

 If you are going to give tests then at least make them so they will reflect what a developer can do, if not don't bother. Scores may look nice but they don't really show what  a developer really knows about developing.

 Simmy

Hi Simmy,

Thanks for your feedback! We will look into the Access 2003 test and the improvements that you suggest.

Thanks,

Sawan
 

Whay aren't there any tests for VS 2005 technologies and languages?

Hi Simmy,

We have a test for Visual Studio: 

https://my.odesk.com/console/g/view_test.php?recno=579&tot=3&pos=2

There are also other tests for Microsoft technologies, please check in the "Microsoft technologies" test category.

Thanks,

Sawan

That test looks like it covers a very wide swath of visual studio.  Also no where does it say which version of VS it is for. Don't you think it would be better to focus on a single language? For me the test is useless since I am a VB developer but not a VC++ developer. (Although based on the test I've taken I can probably answer the VB questions only and still get a passing score)

 Also you have a VB 6 and a VB test, they look the same and only one is included under MS Technolgies. Which BTW only has 3 tests when there are clearly a number of others that should be included (eg. ADO .NET)

Simmy  

Simmy,

 We do have a test for ADO.NET:

https://my.odesk.com/console/g/view_test.php?recno=470&tot=5&pos=0

We also have other Microsoft tests for

- DOT NET fundamentals

- VB.NET

Those tests were included in the 'General Programming' category. We have now recategorized them in the 'Microsoft Technologies' category, so please have a look at them (and of course take them! ).

Thanks for your great feedback!

--Sawan

Irfan, your feedback is highly appreciated (as usual ).

 My replies on each of them in bold below, please take a look.

Also, would like to get your thoughts on an issue:

Right now, lot of providers are taking tests, but not making the test scores "public" (even the ones who are getting good scores). Currently, the default status for the any test that is taken is "private". We are thinking of changing the  defaulting logic to: if test-taker passes the test, make the test score public by default. If not, keep it "private"

 Thoughts? 

 1- It would be a better if we could go  to any question in the test from a "list view", maybe leave some questions unanswered initially and return back if we have more time to finish the test. I guess this is a design criteria and you would possibly not like to implement it, but I strongly believe that all test takers like me would like to use such a feature

We have intentionally disabled that ability to go back and answer questions, essentialy to maintain the integrity of the tests. Giving this ability would allow a  test-taker the opportuntiy and time to go search the answer on the internet or in a book and and then come back and answer the questions for which he/she doesnt know the answers to himself/herself.

 
2- The test duration shows on each page but it only updates on the next test page. It would be nice if a simple dynamic JavaScript clock is shown on each page and even warn the user if you've spent too much time on a specific question and have forgotten to proceed to the next one. For ex if there are 40 questions and 40 min's of test time, the clock could go flashing in red color if you've spent more than 1 min on each page 40 quest/40 mins = 1 min/question

Excellent suggestion, I am going to pass onto the Enginnering team to see if it can implemented. 

3- As I've also written on the survey some pages had "character encoding" problems. I use the English version of Windows XP Profession
with Service Pack 2 [with latest updates] and my browser is also in English : Firefox 1.5.0.4  and it is up to date as of today. I can
provide some screenshots as I did not have an option to attach files in the survey.

Yes, known issue. We are working on it.

 4- If there were more explanations in the scoring system, it would have been better. Some questions have multiple correct answers with
checkboxes . For ex if a & b & c is true from the answers a,b,c,d and if I select a & b only [leaving c unselected which is
also true], do I get a "partial score" for selecting 2 of the 3 correct answers or the answer is evaluated completely false ?

Good catch: there are no partial points for partially correct answers. I have updated the FAQ to include that info:

online_testing_FAQ#A10

5- There are some problems with the stats, you may be aware of them but let me write below in case there's something new for you :
- The
PHP4 test I've taken [Test ID: 158] : There were 10 test takers and only 6 of them have qualified in the test [at that time]. Somehow my
rank showed 11th. If it shows my rank according to all of the test takers among all tests, it's not relevant.
I and any buyer would be
interested in to know the rank relevant with the test [i.e. ranked 3rd in 10 test takes]. If it is showing my rank from this test only, the
data is wrong as only 10 people have taken this test, my rank can't be 11.

Unable to reproduce the issue, since the #'s you quote have changed in the mean time

https://my.odesk.com/console/g/view_exam.php?recno=158

Currently, you are ranked 9th out of 31 test takers so far, so it seems t be fine.  Is there another test where you can point the issue?

 "Test results" detail screen
    - "Score By Areas" section shows rows like  "5. Include Files      50%"
   
Better is to show if possible with the amount of  questions for the area. For this example, I've scored 50% from include files. It matters
a lot if I've given 1 correct answer out of 2 questions or I've given 5 correct answers out of 10 questions. It would be nice to show the total
number of questions related with the area field.

Good catch, I am passing it on to the Engg team.

 6- The default view for "My Tests" show only test that are made public by the developer. You have to re-search by selecting "All"  from status drop-down in order to see all tests. I do not think that a
developer will take 100's of test so from usability point of view, it'll be a lot easier to set the default view to "All" so that
developers can see all tests they've taken regardless of the status field.
This would also decrease the server load/traffic on the
server as I presume most of the coders will prefer to see all tests together so they’ll be performing an extra step to see all tests.

We are hoping that providers will be taking a lot of tests, hence the default to "public".

7- If developers can see next date that they can re-take the test before the test details screen, on the test list page, it would be
better [having 1 less click] to see that they can re-take the test.

Yes, good catch: passing it on to the engg team.

Irfan, your feedback is highly appreciated (as usual ).

 My replies on each of them in bold below, please take a look.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Hi Sawan,
I'm happy to see that you've found my suggestions useful. That's really what I like most about oDesk, you and all oDesk people are really active and collaborate a lot
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Also, would like to get your thoughts on an issue:

Right now, lot of providers are taking tests, but not making the
test scores "public" (even the ones who are getting good scores).
Currently, the default status for the any test that is taken is
"private". We are thinking of changing the  defaulting logic to: if
test-taker passes the test, make the test score public by default. If
not, keep it "private"

 Thoughts?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Well my vote would be keeping the default behaviour as the other way could be a bit worse. There could not be much harm keeping a test result private as default which the provider can make public anytime but the other way would be a bit more like forcing the providers, something like intervening their preferences. It could be that although the score was good, the coder thought he could get a better score and would like to try again later after the time limit. And as with most coders, we have a pretty tight schedule and postpone it later when the time comes you can re-take the test as you'd be busy with many things on that time.

Here's what I'd do if I were you :
- Just spot about 10-15 developers who had good passing scores but kept them private and ask them the reason for doing so . There could be other reasons we can never think of as well.
- After the test is taken and if the test score is a passing/good one, have a javascript dialogue box or even a DHTML popup appear automatically on the results page, write a good explanation/description there and have the users close the jscript dialog box/DHTML popup by  choosing the  buttons :
 [Keep the Test Result Private]  [Make the Test Result Public]
This way you'd have the coder draw their attention to this point and possibly get more public results maybe ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

 1- It would be a better if we could go  to any question in the test
from a "list view", maybe leave some questions unanswered initially and
return back if we have more time to finish the test. I guess this is a
design criteria and you would possibly not like to implement it, but I
strongly believe that all test takers like me would like to use such a
feature

We have intentionally disabled that ability to go back and
answer questions, essentialy to maintain the integrity of the tests.
Giving this ability would allow a  test-taker the opportuntiy and time
to go search the answer on the internet or in a book and and then come
back and answer the questions for which he/she doesnt know the answers
to himself/herself.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Well that's definitely a good point but a coder can always look for a reference by opening a second browser window anytime. Maybe you could decrease the total time for the tests a bit but sometimes you're not sure with a questions and would like to think over it -if you have surplus time after the quesions are finished-  so this feature really makes sense to me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
 
2- The test duration shows on each page but it only updates on
the next test page. It would be nice if a simple dynamic JavaScript
clock is shown on each page and even warn the user if you've spent too
much time on a specific question and have forgotten to proceed to the
next one. For ex if there are 40 questions and 40 min's of test time,
the clock could go flashing in red color if you've spent more than 1
min on each page 40 quest/40 mins = 1 min/question

Excellent suggestion, I am going to pass onto the Enginnering team to see if it can implemented. 

3- As I've also written on the survey some pages had "character
encoding" problems. I use the English version of Windows XP Profession
with Service Pack 2 [with latest updates] and my browser is also in
English : Firefox 1.5.0.4  and it is up to date as of today. I can
provide some screenshots as I did not have an option to attach files in the survey.

Yes, known issue. We are working on it.

 4- If there were more explanations in the scoring system, it would
have been better. Some questions have multiple correct answers with
checkboxes . For ex if a & b & c is true from the answers
a,b,c,d and if I select a & b only [leaving c unselected which is
also true], do I get a "partial score" for selecting 2 of the 3 correct answers or the answer is evaluated completely false ?

Good catch: there are no partial points for partially correct answers. I have updated the FAQ to include that info:

online_testing_FAQ#A10

5- There are some problems with the stats, you may be aware of them
but let me write below in case there's something new for you :
- The
PHP4 test I've taken [Test ID: 158] : There were 10 test takers and
only 6 of them have qualified in the test [at that time]. Somehow my
rank showed 11th. If it shows my rank according to all of the test takers among all tests, it's not relevant.
I and any buyer would be interested in to know the rank relevant with the test [i.e. ranked 3rd
in 10 test takers]. If it is showing my rank from this test only, the
data is wrong as only 10 people have taken this test, my rank can't be 11.

Unable to reproduce the issue, since the #'s you quote have changed in the mean time

https://my.odesk.com/console/g/view_exam.php?recno=158

Currently, you are ranked 9th out of 31 test takers so far,
so it seems t be fine.  Is there another test where you can point the
issue?

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Well that was the only one case I had during beta tests, so I'm afraid I can not provide another sample
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

 "Test results" detail screen
    - "Score By Areas" section shows rows like  "5. Include Files      50%"
   
Better is to show if possible with the amount of  questions for the
area. For this example, I've scored 50% from include files. It matters
a lot if I've given 1 correct answer out of 2 questions or I've given 5
correct answers out of 10 questions. It would be nice to show the total
number of questions related with the area field.

Good catch, I am passing it on to the Engg team.

 6- The default view for "My Tests" show only test that are made
public by the developer. You have to re-search by selecting "All"  from
status drop-down in order to see all tests. I do not think that a
developer will take 100's of test so from usability point of view,
it'll be a lot easier to set the default view to "All" so that
developers can see all tests they've taken regardless of the status field.
This would also decrease the server load/traffic on the server as I presume most of the coders will prefer to see all tests
together so they’ll be performing an extra step to see all tests.

We are hoping that providers will be taking a lot of tests, hence the default to "public".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Yes but I'm not proposing the default view to private either :)  Given a typical skilled developer's skills he'll take about 20-30 tests at most. Keeping all the tests [both public on private] on "My Tests"
would be more neat and organized for them. It really helps to see all results on one page.
The 'private' marked tests will be mostly the ones who did not get a passing grade or had a low score and coders are most likely to re-take this tests after the time limit. So they will mostly be selecting "All" from
status and refresh this page almost all the time to see if the time limit had expired.

I've got also 1 suggestion here. The list view shows ranks here which is nice but seeing the overall would also be nice. Instead of showing "9" as the rank showing it as 9/19/32 rank/qualified/all test takers would be nice
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

7- If developers can see next date that they can re-take the test
before the test details screen, on the test list page, it would be better [having 1 less click] to see that they can re-take the test.

Yes, good catch: passing it on to the engg team.